Parallel computing techniques

Parallel computing means to divide a job into several tasks and use more than one processor simultaneously to perform these tasks. Assume you have developed a new estimation method for the parameters of a complicated statistical model. After you prove the asymptotic characteristics of the method (for instance, asymptotic distribution of the estimator), you wish to perform many simulations to assure the goodness of the method for reasonable numbers of data values and for different values of parameters. You must generate simulated data, for example, 100 000 times for each length and parameter value. The total simulation work requires a huge number of random number generations and takes a long time on your PC. If you use 100 PCs in your institute to run these simulations simultaneously, you may expect that the total execution time will be 1/100. This is the simple idea of parallel computing.

[1]  Anthony Skjellum,et al.  Using MPI - portable parallel programming with the message-parsing interface , 1994 .

[2]  Quentin F. Stout,et al.  A program for sequential allocation of Bernoulli populations , 1999 .

[3]  Andrew S. Tanenbaum,et al.  Modern operating systems, 2nd Edition , 2001 .

[4]  Michael Mascagni.  SPRNG: A Scalable Library for Pseudorandom Number Generation , 1999, PPSC.

[5]  Michael J. Flynn,et al.  Very high-speed computing systems , 1966 .

[6]  Joël M. Malard.  Parallel restricted maximum likelihood estimation for linear models with a dense exogenous matrix , 2002, Parallel Comput..

[7]  Al Geist,et al.  PVM (Parallel Virtual Machine) , 2011, Encyclopedia of Parallel Computing.

[8]  G. Amdhal,et al.  Validity of the single processor approach to achieving large scale computing capabilities , 1967, AFIPS '67 (Spring).

[9]  Rohit Chandra,et al.  Parallel programming in openMP , 2000 .

[10]  D. Parkinson,et al.  A parallel implementation of the maximum likelihood method in positron emission tomography image reconstruction , 1999 .

[11]  Benjamin Ray Seyfarth,et al.  How to Build a Beowulf: A Guide to the Implementation and Application of PC Clusters , 2000, Scalable Comput. Pract. Exp..

[12]  David R. Butenhof.  Programming with POSIX threads , 1993 .

[13]  Yoshikazu Yamamoto,et al.  A statistical package based on Pnuts , 2000 .

[14]  Nicos Christofides,et al.  The optimal discretization of probability density functions , 1999 .

[15]  Friedrich Leisch,et al.  Editorial Porting R to Darwin/x11 and Mac Os X Mac Os X Application Environments User Experience Porting Problems Rpvm: Cluster Statistical Computing in R , 2022 .

[16]  Yoshikazu Yamamoto,et al.  DISTRIBUTED PROCESSING FUNCTIONS OF A TIME SERIES ANALYSIS SYSTEM , 2002 .

[17]  John L. Gustafson,et al.  Reevaluating Amdahl's law , 1988, CACM.

[18]  Na Li,et al.  Simple Parallel Statistical Computing in R , 2007 .

[19]  Los Angeles,et al.  Applications of Parallel Programming in Statistics , 2002 .

[20]  Markus Hegland,et al.  A parallel solver for generalised additive models , 1999 .

[21]  Graham D. Riley,et al.  Parallel implementation of a multilevel modelling package , 1999 .

[22]  Anthony Skjellum,et al.  Using MPI: portable parallel programming with the message-passing interface, 2nd Edition , 1999, Scientific and engineering computation series.

[23]  Mark J. Schervish.  Applications of Parallel Computation to Statistical Inference , 1988 .

[24]  Scott Oaks,et al.  Java Threads , 1997 .

[25]  Douglas M. Bates,et al.  Programming With Data: A Guide to the S Language , 1999, Technometrics.

[26]  J. G. Bethlehem,et al.  COMPSTAT 2000, Proceedings in Computational Statistics , 2000 .

[27]  John A. Rice,et al.  A multi-threaded extension to a high level interactive statistical computing environment , 1997 .

[28]  Robert B. Ross,et al.  Using MPI-2: Advanced Features of the Message Passing Interface , 2003, CLUSTER.

[29]  Ilkka Tuomi,et al.  The Lives and Death of Moore's Law , 2002, First Monday.

[30]  Jack Dongarra,et al.  PVM: Parallel virtual machine: a users' guide and tutorial for networked parallel computing , 1995 .

[31]  Jack Dongarra,et al.  ScaLAPACK Users' Guide , 1987 .

[32]  Jeffrey S. Racine,et al.  Parallel distributed kernel estimation , 2002 .

[33]  Guy L. Steele,et al.  The High Performance Fortran Handbook , 1993 .

[34]  Christopher A. Swann,et al.  Maximum Likelihood Estimation Using Parallel Computing: An Introduction to MPI , 2002 .

[35]  Maurice Clint,et al.  Re-engineering statistical software for efficient parallel execution , 1999 .