Recent approaches to global optimization problems through Particle Swarm Optimization

This paper presents an overview of our most recent results concerning the Particle Swarm Optimization (PSO) method. Techniques for the alleviation of local minima, and for detecting multiple minimizers are described. Moreover, results on the ability of the PSO in tackling Multiobjective, Minimax, Integer Programming and ℓ1 errors-in-variables problems, as well as problems in noisy and continuously changing environments, are reported. Finally, a Composite PSO, in which the heuristic parameters of PSO are controlled by a Differential Evolution algorithm during the optimization, is described, and results for many well-known and widely used test functions are given.

[1]  E. O. Roxin.  Optimization Theory and Applications (Lamberto Cesari) , 1984 .

[2]  Elijah Polak,et al.  Optimization: Algorithms and Consistent Approximations , 1997 .

[3]  Hans-Paul Schwefel,et al.  Numerical Optimization of Computer Models , 1982 .

[4]  Eckart Zitzler,et al.  Evolutionary algorithms for multiobjective optimization: methods and applications , 1999 .

[5]  V. F. Demʹi︠a︡nov,et al.  Introduction to minimax , 1976 .

[6]  A. V. Levy,et al.  Topics in global optimization , 1982 .

[7]  M. N. Vrahatis,et al.  A New Unconstrained Optimization Method for Imprecise Function and Gradient Values , 1996 .

[8]  Peter J. Angeline,et al.  Tracking Extrema in Dynamic Environments , 1997, Evolutionary Programming.

[9]  David B. Fogel,et al.  Evolutionary Computation: Towards a New Philosophy of Machine Intelligence , 1995 .

[10]  Ralf Salomon,et al.  Evolutionary algorithms and gradient search: similarities and differences , 1998, IEEE Trans. Evol. Comput..

[11]  Singiresu S. Rao.  Engineering Optimization : Theory and Practice , 2010 .

[12]  Hans-Paul Schwefel,et al.  Evolution and optimum seeking , 1995, Sixth-generation computer technology series.

[13]  Günter Rudolph,et al.  An Evolutionary Algorithm for Integer Programming , 1994, PPSN.

[14]  G. A. Watson,et al.  On the solution of the errors in variables problem using thel1 norm , 1991 .

[15]  K. Parsopoulos,et al.  Stretching technique for obtaining global minimizers through Particle Swarm Optimization , 2001 .

[16]  Xing-Si Li,et al.  AN AGGREGATE FUNCTION METHOD FOR NONLINEAR PROGRAMMING , 1991 .

[17]  R. W. Dobbins,et al.  Computational intelligence PC tools , 1996 .

[18]  W. Murray,et al.  A Projected Lagrangian Algorithm for Nonlinear Minimax Optimization , 1980 .

[19]  John E. Dennis,et al.  Multidirectional search: a direct search algorithm for parallel machines , 1989 .

[20]  Leon S. Lasdon,et al.  Optimization in engineering design , 1967 .

[21]  Ji-Ming Peng,et al.  A non-interior continuation method for generalized linear complementarity problems , 1999, Math. Program..

[22]  Zbigniew Michalewicz,et al.  Genetic Algorithms + Data Structures = Evolution Programs , 1992, Artificial Intelligence.

[23]  Sandro Ridella,et al.  Minimizing multimodal functions of continuous variables with the “simulated annealing” algorithmCorrigenda for this article is available here , 1987, TOMS.

[24]  William T. Reeves.  Particle systems—a technique for modeling a class of fuzzy objects , 1993 .

[25]  Nikolaus Hansen,et al.  Verallgemeinerte individuelle Schrittweitenregelung in der Evolutionsstrategie , 1998 .

[26]  V. J. Torczoit,et al.  Multidirectional search: a direct search algorithm for parallel machines , 1989 .

[27]  John A. Nelder,et al.  A Simplex Method for Function Minimization , 1965, Comput. J..

[28]  E. Polak.  On the mathematical foundations of nondifferentiable optimization in engineering design , 1987 .

[29]  P. Pardalos,et al.  Minimax and applications , 1995 .

[30]  Hans-Paul Schwefel,et al.  Evolution strategies – A comprehensive introduction , 2002, Natural Computing.

[31]  Russell C. Eberhart,et al.  Parameter Selection in Particle Swarm Optimization , 1998, Evolutionary Programming.

[32]  M. N. Vrahatis,et al.  Particle swarm optimization method in multiobjective problems , 2002, SAC '02.

[33]  Michael N. Vrahatis,et al.  Particle swarm optimization for minimax problems , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[34]  H. Beyer.  Evolutionary algorithms in noisy environments : theoretical issues and guidelines for practice , 2000 .

[35]  Hans-Georg Beyer,et al.  The Theory of Evolution Strategies , 2001, Natural Computing Series.

[36]  E. T. Jaynes,et al.  Where do we Stand on Maximum Entropy , 1979 .

[37]  R. Hodgson,et al.  Genetic Algorithm Approach to Particle Identification by Light Scattering. , 2000, Journal of colloid and interface science.

[38]  J. Spall.  Implementation of the simultaneous perturbation algorithm for stochastic optimization , 1998 .

[39]  D. E. Goldberg,et al.  Genetic Algorithms in Search , 1989 .

[40]  Russell C. Eberhart,et al.  Comparison between Genetic Algorithms and Particle Swarm Optimization , 1998, Evolutionary Programming.

[41]  Jorge Nocedal,et al.  Theory of algorithms for unconstrained optimization , 1992, Acta Numerica.

[42]  Michael N. Vrahatis,et al.  Computing with certainty individual members of families of periodic orbits of a given period , 2001 .

[43]  M. E. Muller,et al.  A Note on the Generation of Random Normal Deviates , 1958 .

[44]  Thomas Bäck,et al.  Evolutionary Algorithms in Theory and Practice , 1996 .

[45]  Robert Hooke,et al.  `` Direct Search'' Solution of Numerical and Statistical Problems , 1961, JACM.

[46]  Jorge J. Moré,et al.  Testing Unconstrained Optimization Software , 1981, TOMS.

[47]  A. Skinner,et al.  Neural networks in computational materials science: training algorithms , 1995 .

[48]  Jean-Baptiste Hiriart-Urruty,et al.  On optimality conditions in nondifferentiable programming , 1978, Math. Program..

[49]  D A Pierre,et al.  Optimization Theory with Applications , 1986 .

[50]  Marimuthu Palaniswami,et al.  Computational Intelligence: A Dynamic System Perspective , 1995 .

[51]  Carl Tim Kelley,et al.  Iterative methods for optimization , 1999, Frontiers in applied mathematics.

[52]  Russell C. Eberhart,et al.  A new optimizer using particle swarm theory , 1995, MHS'95. Proceedings of the Sixth International Symposium on Micro Machine and Human Science.

[53]  Thomas Bäck,et al.  Evolutionary algorithms in theory and practice - evolution strategies, evolutionary programming, genetic algorithms , 1996 .

[54]  Lothar Thiele,et al.  Comparison of Multiobjective Evolutionary Algorithms: Empirical Results , 2000, Evolutionary Computation.

[55]  J. Snyman,et al.  A multi-start global minimization algorithm with dynamic search trajectories , 1987 .

[56]  Craig W. Reynolds.  Flocks, herds, and schools: a distributed behavioral model , 1998 .

[57]  Saul Krasner,et al.  The Ubiquity of chaos , 1990 .

[58]  Song Xu,et al.  A non–interior predictor–corrector path following algorithm for the monotone linear complementarity problem , 2000, Math. Program..

[59]  Wolfgang Banzhaf,et al.  Genetic Programming: An Introduction , 1997 .

[60]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[61]  R. Kelahan,et al.  Application of the adaptive random search to discrete and mixed integer optimization , 1978 .

[62]  John R. Koza,et al.  Genetic programming - on the programming of computers by means of natural selection , 1993, Complex adaptive systems.

[63]  E. K. Blum,et al.  Approximation of Boolean Functions by Sigmoidal Networks: Part I: XOR and Other Two-Variable Functions , 1989, Neural Computation.

[64]  C. Charalambous,et al.  Nonlinear programming using minimax techniques , 1974 .

[65]  G. Alistair Watson,et al.  An Algorithm for Minimax Approximation in the Nonlinear Case , 1969, Comput. J..

[66]  P. Pardalos,et al.  Handbook of global optimization , 1995 .

[67]  Konstantinos E. Parsopoulos,et al.  PARTICLE SWARM OPTIMIZER IN NOISY AND CONTINUOUSLY CHANGING ENVIRONMENTS , 2001 .

[68]  Yaochu Jin,et al.  Dynamic Weighted Aggregation for evolutionary multi-objective optimization: why does it work and how? , 2001 .

[69]  G. A. Watson,et al.  Choice of norms for data fitting and function approximation , 1998, Acta Numerica.

[70]  Michael N. Vrahatis,et al.  Particle swarm optimization for integer programming , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[71]  D. Bertsekas,et al.  A new penalty function method for constrained minimization , 1972, CDC 1972.

[72]  W. Paszkowicz,et al.  Application of the Smooth Genetic Algorithm for Indexing Powder Patterns – Tests for the Orthorhombic System , 1996 .

[73]  János D. Pintér,et al.  Global optimization in action , 1995 .

[74]  Michael N. Vrahatis,et al.  Artificial nonmonotonic neural networks , 2001, Artif. Intell..

[75]  D. Bertsekas.  Approximation procedures based on the method of multipliers , 1977 .

[76]  Drossos,et al.  Method for computing long periodic orbits of dynamical systems. , 1996, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[77]  M. N. Vrahatis,et al.  Objective function “stretching” to alleviate convergence to local minima , 2001 .

[78]  Aimo A. Törn,et al.  Global Optimization , 1999, Science.

[79]  Zbigniew Michalewicz,et al.  Handbook of Evolutionary Computation , 1997 .

[80]  Peter Nordin,et al.  Genetic programming - An Introduction: On the Automatic Evolution of Computer Programs and Its Applications , 1998 .

[81]  Francisco Facchinei,et al.  A smoothing method for mathematical programs with equilibrium constraints , 1999, Math. Program..

[82]  A. Neumaier,et al.  A grid algorithm for bound constrained optimization of noisy functions , 1995 .

[83]  Zbigniew Michalewicz,et al.  Genetic Algorithms Plus Data Structures Equals Evolution Programs , 1994 .

[84]  R. Eberhart,et al.  Comparing inertia weights and constriction factors in particle swarm optimization , 2000, Proceedings of the 2000 Congress on Evolutionary Computation. CEC00 (Cat. No.00TH8512).

[85]  A. Conn,et al.  An Efficient Method to Solve the Minimax Problem Directly , 1978 .

[86]  Song Xu,et al.  Smoothing Method for Minimax Problems , 2001, Comput. Optim. Appl..

[87]  V. F. Demʹi︠a︡nov,et al.  Introduction to minimax , 1976 .

[88]  Virginia Torczon,et al.  On the Convergence of the Multidirectional Search Algorithm , 1991, SIAM J. Optim..

[89]  Michael N. Vrahatis,et al.  Application of the Characteristic Bisection Method for locating and computing periodic orbits in molecular systems , 2001 .

[90]  H. Britt,et al.  The Estimation of Parameters in Nonlinear, Implicit Models , 1973 .

[91]  Peter J. Angeline,et al.  Evolutionary Optimization Versus Particle Swarm Optimization: Philosophy and Performance Differences , 1998, Evolutionary Programming.

[92]  M. Fukushima,et al.  Minimizing multimodal functions by simplex coding genetic algorithm , 2003 .

[93]  Günter Rudolph,et al.  Contemporary Evolution Strategies , 1995, ECAL.

[94]  James Kennedy,et al.  Particle swarm optimization , 1995, Proceedings of ICNN'95 - International Conference on Neural Networks.

[95]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[96]  James C. Spall,et al.  AN OVERVIEW OF THE SIMULTANEOUS PERTURBATION METHOD FOR EFFICIENT OPTIMIZATION , 1998 .

[97]  Hans-Paul Schwefel,et al.  Evolution and Optimum Seeking: The Sixth Generation , 1993 .

[98]  Ulf Grenander,et al.  A stochastic nonlinear model for coordinated bird flocks , 1990 .

[99]  E. Wilson,et al.  Sociobiology: The New Synthesis , 1975 .

[100]  Nikolaus Hansen,et al.  Completely Derandomized Self-Adaptation in Evolution Strategies , 2001, Evolutionary Computation.

[101]  Mark M. Millonas,et al.  Swarms, Phase Transitions, and Collective Intelligence , 1993, adap-org/9306002.

[102]  G. Hogg,et al.  UNCONSTRAINED DISCRETE NONLINEAR PROGRAMMING , 1979 .

[103]  Dimitri P. Bertsekas,et al.  A new algorithm for solution of resistive networks involving diodes , 1976 .

[104]  Timothy S Bush,et al.  Evolutionary programming techniques for predicting inorganic crystal structures , 1995 .

[105]  James Kennedy,et al.  The Behavior of Particles , 1998, Evolutionary Programming.

[106]  R. Horst,et al.  Global Optimization: Deterministic Approaches , 1992 .

[107]  M. N. Vrahatisa,et al.  A class of gradient unconstrained minimization algorithms with adaptive stepsize , 1999 .

[108]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[109]  J. Spall.  Multivariate stochastic approximation using a simultaneous perturbation gradient approximation , 1992 .

[110]  M. Powell.  A Direct Search Optimization Method That Models the Objective and Constraint Functions by Linear Interpolation , 1994 .

[111]  Michael N. Vrahatis,et al.  Modification of the Particle Swarm Optimizer for Locating All the Global Minima , 2001 .

[112]  Vijay K. Garg,et al.  A genetic algorithm for fitting Lorentzian line shapes in Mössbauer spectra , 1997 .

[113]  A. Neumaier,et al.  Solving minimax problems by interval methods , 1990 .

[114]  David W. Corne,et al.  Approximating the Nondominated Front Using the Pareto Archived Evolution Strategy , 2000, Evolutionary Computation.

[115]  Vassilis P. Plagianakos,et al.  Training neural networks with threshold activation functions and constrained integer weights , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[116]  Z. Kam,et al.  Absorption and Scattering of Light by Small Particles , 1998 .

[117]  Vassilis P. Plagianakos,et al.  Improving the Particle Swarm Optimizer by Function “Stretching” , 2001 .

[118]  Eldon Hansen,et al.  Global optimization using interval analysis , 1992, Pure and applied mathematics.

[119]  Yue Shi,et al.  A modified particle swarm optimizer , 1998, 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98TH8360).

[120]  Michael N. Vrahatis,et al.  PARTICLE SWARM OPTIMIZATION FOR IMPRECISE PROBLEMS , 2002 .

[121]  G. Alistair Watson,et al.  The use of the L 1 norm in nonlinear errors-in-variables problems , 1997 .

[122]  C. T. Kelley,et al.  An Implicit Filtering Algorithm for Optimization of Functions with Many Local Minima , 1995, SIAM J. Optim..

[123]  Sam Kwong,et al.  Genetic algorithms and their applications , 1996, IEEE Signal Process. Mag..