A View of the Limitations, Opportunities, and Challenges in Parallel Nonlinear Optimization

Abstract The availability and power of parallel and distributed computers is having a significant impact on how expensive problems are solved in all areas of numerical computation, and is likely to have an even larger impact in the future. This paper presents a view of how the consideration of parallelism is affecting, and is likely to affect, one important field within numerical computation, the field of nonlinear optimization. It does not attempt to survey the research that has been done in parallel nonlinear optimization. Rather it presents a set of examples, drawn mainly from our own research, that illustrate many of the limitations, opportunities, and challenges inherent in incorporating parallelism into the field of nonlinear optimization. These examples include parallel methods for unconstrained optimization problems with a small to moderate number of variables, parallel methods for large block bordered systems of nonlinear equations, and parallel methods for small-scale and large-scale global optimization problems. Our overall conclusions are mixed. For most generic optimization problems with a small to moderate number of variables, the consideration of parallelism does not appear to be leading to major algorithmic innovations. For many classes of large-scale problems, however, the consideration of parallelism appears to be creating opportunities for the development of interesting new methods that may be advantageous on parallel and sometimes even on sequential computers. In addition, a number of large-scale parallel optimization algorithms exhibit irregular, coarse-grain structure, which leads to interesting computer science challenges in areas such as dynamic scheduling and load-balancing.

[1]  Jorge J. Moré,et al.  Testing Unconstrained Optimization Software , 1981, TOMS.

[2]  Joel H. Saltz,et al.  Unstructured scientific computation on scalable multiprocessors , 1992 .

[3]  Robert B. Schnabel Parallel Nonlinear Optimization: Limitations, Opportunities, and Challenges , 1994 .

[4]  K. Tanabe,et al.  Mathematical Programming: Recent Developments and Applications. , 1990 .

[5]  J. Dennis,et al.  Direct Search Methods on Parallel Machines , 1991 .

[6]  J. Northby Structure and binding of Lennard‐Jones clusters: 13≤N≤147 , 1987 .

[7]  Robert B. Schnabel,et al.  Adaptive, Asynchronous Stochastic Global Optimization Algorithms for Sequential and Parallel Computation , 1989 .

[8]  A. E. Fincham,et al.  Parallel Computation , 1999, Algorithms and Theory of Computation Handbook.

[9]  M. J. Quinn,et al.  Parallel Computing: Theory and Practice , 1994 .

[10]  John E. Dennis,et al.  Problem Formulation for Multidisciplinary Optimization , 1994, SIAM J. Optim..

[11]  André van der Hoek,et al.  A Parallel Global Optimization Method for Solving Molecular Cluster and Polymer Conformation Problems , 1995, PPSC.

[12]  Richard H. Byrd,et al.  A New Large-Scale Global Optimization Method and Its Application to Lennard-Jones Problems ; CU-CS-630-92 , 1992 .

[13]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[14]  George L. Nemhauser,et al.  Handbooks in operations research and management science , 1989 .

[15]  Petter E. Bjørstad,et al.  Proceedings of the Seventh SIAM Conference on Parallel Processing for Scientific Computing, PPSC 1995, San Francisco, California, USA, February 15-17, 1995 , 1995, PPSC.

[16]  Alexander H. G. Rinnooy Kan,et al.  Stochastic methods for global optimization , 1984 .

[17]  Richard H. Byrd,et al.  Parallel Global Optimization Methods for Molecular Configuration Problems , 1993, SIAM Conference on Parallel Processing for Scientific Computing.

[18]  Thomas F. Coleman,et al.  Large-Scale Numerical Optimization , 1990 .

[19]  Robert B. Schnabel,et al.  Globally convergent parallel algorithms for solving block bordered systems of nonlinear equations , 1992 .

[20]  Gene H. Golub,et al.  Scientific computing: an introduction with parallel computing , 1993 .

[21]  R. Fletcher Practical Methods of Optimization , 1988 .

[22]  W. Hager,et al.  Large Scale Optimization : State of the Art , 1993 .

[23]  J. Ben Rosen,et al.  A discrete-continuous algorithm for molecular energy minimization , 1992, Proceedings Supercomputing '92.

[24]  D. Coker,et al.  Structure and vibrational spectroscopy of the water dimer using quantum simulation , 1987 .

[25]  Allan Gottlieb,et al.  Highly parallel computing , 1989, Benjamin/Cummings Series in computer science and engineering.

[26]  G. R. Shubin,et al.  A comparison of optimization-based approaches for a model computational aerodynamics design problem , 1992 .

[27]  Richard H. Byrd,et al.  Using parallel function evaluations to improve hessian approximation for unconstrained optimization , 1988 .

[28]  Robert B. Schnabel,et al.  Centralized and Distributed Dynamic Scheduling for Adaptive, Parallel Algorithms ; CU-CS-516-91 , 1991 .

[29]  Richard H. Byrd,et al.  Parallel Methods for Solving Nonlinear Block Bordered Systems of Equations , 1989, SIAM J. Sci. Comput..

[30]  Iain S. Duff,et al.  On Permutations to Block Triangular Form , 1977 .

[31]  Richard H. Byrd,et al.  Parallel quasi-Newton methods for unconstrained optimization , 1988, Math. Program..

[32]  R. Horst,et al.  Global Optimization: Deterministic Approaches , 1992 .

[33]  Robert B. Schnabel,et al.  Dynamic Scheduling Strategies for an Adaptive, Asynchronous Parallel Global Optimization Algorithm ; CU-CS-625-92 , 1992 .

[34]  Elizabeth Eskow,et al.  Mathematical modeling of a parallel global optimization algorithm , 1989, Parallel Comput..

[35]  José Mario Martínez,et al.  Triangular Decomposition Methods for Solving Reducible Nonlinear Systems of Equations , 1994, SIAM J. Optim..

[36]  Stephen G. Nash,et al.  Block truncated-Newton methods for parallel optimization , 1989, Math. Program..

[37]  Peter J. M. van Laarhoven,et al.  Parallel variable metric algorithms for unconstrained optimization , 1985, Math. Program..

[38]  Stephen G. Nash,et al.  A General-Purpose Parallel Algorithm for Unconstrained Optimization , 1991, SIAM J. Optim..

[39]  R. Byrd,et al.  A New Stochastic/Perturbation Method for Large-Scale Global Optimization and its Application to Water Cluster Problems , 1994 .

[40]  Alexander H. G. Rinnooy Kan,et al.  Concurrent stochastic methods for global optimization , 1990, Math. Program..

[41]  Philip E. Gill,et al.  Practical optimization , 1981 .

[42]  Thomas F. Coleman,et al.  A parallel build-up algorithm for global energy minimizations of molecular clusters using effective energy simulated annealing , 1993, J. Glob. Optim..

[43]  E. Spedicato Algorithms for continuous optimization : the state of the art , 1994 .