Hybrid Coupled Local Minimizers

This paper proposes an improved global optimization technique, named Hybrid Coupled Local Minimizers (HCLM), which is inspired on the method of coupled local minimizers (CLM). The HCLM method uses a set of search points, called “particles”, initially spread over the search space, that are occasionally impulsively coupled, instead of permanently coupled. This approach leads to a much better optimization efficiency, because it combines the fast convergence (due to a Newton-based method that is used for each particle) with the capability of global optimization (resulting from the hybrid interaction which enables a parallel strategy). Such parallel and distributed computing process embedded with a trust region strategy relies on the hybrid interconnections of particles rather than individual particles, which is able to fully exploit the parallel nature of the computation and produce results which are more globally optimal. It is worth mentioning that the stability of each particle and the synchronization of all particles are also derived and proved by means of the contraction theory. The HCLM method is illustrated on several test functions with many local minima and applied to a problem of static nonlinear regression with multilayer perceptrons (MLPs) from given noisy measurement data.

[1]  Jean-Jacques E. Slotine,et al.  On Contraction Analysis for Nonlinear Systems Analyzing stability differentially leads to a new perspective on nonlinear dynamic systems , 1999 .

[2]  Johan A. K. Suykens,et al.  Coupled Simulated Annealing , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[3]  W. Haddad,et al.  Finite-time stabilization of nonlinear impulsive dynamical systems☆ , 2008 .

[4]  Johan A. K. Suykens,et al.  Intelligence and Cooperative Search by Coupled Local Minimizers , 2002, Int. J. Bifurc. Chaos.

[5]  Aaas News,et al.  Book Reviews , 1893, Buffalo Medical and Surgical Journal.

[6]  Sohrab Effati,et al.  A new nonlinear neural network for solving quadratic programming problems , 2005, Appl. Math. Comput..

[7]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[8]  Shengwei Zhang,et al.  Lagrange programming neural networks , 1992 .

[9]  Xuyang Lou,et al.  Stability of Coupled Local Minimizers Within the Lagrange Programming Network Framework , 2013, IEEE Transactions on Circuits and Systems I: Regular Papers.

[10]  D. E. Goldberg,et al.  Genetic Algorithms in Search , 1989 .

[11]  Y. P. Zhang,et al.  Stability analysis of impulsive control systems , 2003 .

[12]  James Kennedy,et al.  Particle swarm optimization , 2002, Proceedings of ICNN'95 - International Conference on Neural Networks.

[13]  Youshen Xia,et al.  An Extended Projection Neural Network for Constrained Optimization , 2004, Neural Computation.

[14]  J. Suykens,et al.  Impulsive Synchronization of Chaotic Lur'e Systems by Measurement Feedback , 1998 .

[15]  R. Fletcher,et al.  Practical Methods of Optimization: Fletcher/Practical Methods of Optimization , 2000 .

[16]  Li-Zhi Liao,et al.  A New Projection-Based Neural Network for Constrained Variational Inequalities , 2009, IEEE Transactions on Neural Networks.

[17]  Saman K. Halgamuge,et al.  Self-organizing hierarchical particle swarm optimizer with time-varying acceleration coefficients , 2004, IEEE Transactions on Evolutionary Computation.

[18]  Okan K. Ersoy,et al.  A Novel Evolutionary Global Optimization Algorithm and its Application in Bioinformatics , 2005 .

[19]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[20]  Wassim M. Haddad,et al.  Finite-time stabilization of nonlinear impulsive dynamical systems , 2007, 2007 European Control Conference (ECC).

[21]  V. Lakshmikantham,et al.  Theory of Impulsive Differential Equations , 1989, Series in Modern Applied Mathematics.

[22]  L. Darrell Whitley,et al.  Evaluating Evolutionary Algorithms , 1996, Artif. Intell..

[23]  W. Marsden.  I and J , 2012 .

[24]  Jean-Jacques E. Slotine,et al.  Compositional Contraction Analysis of Resetting Hybrid Systems , 2006, IEEE Transactions on Automatic Control.

[25]  Dimitri P. Bertsekas,et al.  Constrained Optimization and Lagrange Multiplier Methods , 1982 .

[26]  Xiaolin Hu,et al.  A Recurrent Neural Network for Solving a Class of General Variational Inequalities , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[27]  Johan A. K. Suykens,et al.  Global optimization by coupled local minimizers and its application to FE model updating , 2003 .

[28]  Jean-Jacques E. Slotine,et al.  On Contraction Analysis for Non-linear Systems , 1998, Autom..

[29]  Yue Shi,et al.  A modified particle swarm optimizer , 1998, 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98TH8360).

[30]  Wassim M. Haddad,et al.  Impulsive and Hybrid Dynamical Systems: Stability, Dissipativity, and Control , 2006 .