Convergence and error estimates for time-discrete consensus-based optimization algorithms

We present convergence and error estimates of modified versions of the time-discrete consensus-based optimization (CBO) algorithm proposed in Carrillo et al. (ESAIM: Control Optim Calc Var, 2020) for general non-convex functions. In authors’ recent work (Ha et al. in Math Models Meth Appl Sci 30:2417–2444, 2020), rigorous error analysis of a modified version of the first-order consensus-based optimization algorithm proposed in Carrillo et al. (2020) was studied at the particle level without resorting to the kinetic equation via a mean-field limit. However, the error analysis for the corresponding time- discrete algorithm was not done mainly due to lack of discrete analogue of Itô’s stochastic calculus. In this paper, we provide a simple and elementary convergence and error analysis for a general time-discrete consensus-based optimization algorithm, which includes modifications of the three discrete algorithms in Carrillo et al. (2020), two of which are present in Ha et al. (2020). Our analysis provides numerical stability and convergence conditions for the three algorithms, as well as error estimates to the global minimum.

[1]  L. C. Hsu.  A theorem on the asymptotic behavior of a multiple integral , 1948 .

[2]  Yoshiki Kuramoto,et al.  Self-entrainment of a population of coupled non-linear oscillators , 1975 .

[3]  Huzihiro Araki,et al.  International Symposium on Mathematical Problems in Theoretical Physics , 1975 .

[4]  Charles S. Peskin,et al.  Mathematical aspects of heart physiology , 1975 .

[5]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[6]  Yoshiki Kuramoto,et al.  Chemical Oscillations, Waves, and Turbulence , 1984, Springer Series in Synergetics.

[7]  Emile H. L. Aarts,et al.  Simulated Annealing: Theory and Applications , 1987, Mathematics and Its Applications.

[8]  E. Crow,et al.  Lognormal Distributions: Theory and Applications , 1987 .

[9]  Dan Boneh,et al.  On genetic algorithms , 1995, COLT '95.

[10]  James Kennedy,et al.  Particle swarm optimization , 2002, Proceedings of ICNN'95 - International Conference on Neural Networks.

[11]  Amir Dembo,et al.  Large Deviations Techniques and Applications , 1998 .

[12]  A. Czirók,et al.  Collective Motion , 1999, physics/9902023.

[13]  Jürgen Kurths,et al.  Synchronization - A Universal Concept in Nonlinear Sciences , 2001, Cambridge Nonlinear Science Series.

[14]  Jürgen Kurths,et al.  Synchronization: Phase locking and frequency entrainment , 2001 .

[15]  Goldberg,et al.  Genetic algorithms , 1993, Robust Control Systems with Genetic Algorithms.

[16]  Dimitri P. Bertsekas,et al.  Convex Analysis and Optimization , 2003 .

[17]  R. Spigler,et al.  The Kuramoto model: A simple paradigm for synchronization phenomena , 2005 .

[18]  James Kennedy,et al.  Swarm Intelligence , 2012, Lecture Notes in Computer Science.

[19]  Riccardo Poli,et al.  Particle swarm optimization , 1995, Swarm Intelligence.

[20]  S. Smale,et al.  On the mathematics of emergence , 2007 .

[21]  Xin-She Yang,et al.  Nature-Inspired Metaheuristic Algorithms , 2008 .

[22]  Seung-Yeal Ha,et al.  Emergence of time-asymptotic flocking in a stochastic Cucker-Smale system , 2009 .

[23]  Seung-Yeal Ha,et al.  A simple proof of the Cucker-Smale flocking dynamics and mean-field limit , 2009 .

[24]  T. Vicsek,et al.  Collective Motion , 1999, physics/9902023.

[25]  R. Fetecau,et al.  Emergent behaviour in multi-particle systems with non-local interactions , 2013 .

[26]  Sébastien Motsch,et al.  Heterophilious Dynamics Enhances Consensus , 2013, SIAM Rev..

[27]  R. Pinnau,et al.  A consensus-based model for global optimization and its mean-field limit , 2016, 1604.05648.

[28]  Jos'e A. Carrillo,et al.  An analytical framework for a consensus-based global optimization method , 2016, 1602.00220.

[29]  Seung-Yeal Ha,et al.  Emergent dynamics of the Cucker-Smale flocking model and its variants , 2016, 1604.04887.

[30]  Claudia Totzeck,et al.  A Numerical Comparison of Consensus‐Based Global Optimization to other Particle‐based Global Optimization Schemes , 2018 .

[31]  Yuxin Zhao,et al.  Swarm intelligence: past, present and future , 2017, Soft Computing.

[32]  Doheon Kim,et al.  Convergence of a first-order consensus-based global optimization algorithm , 2019, Mathematical Models and Methods in Applied Sciences.

[33]  Seung-Yeal Ha,et al.  Vehicular traffic, crowds, and swarms: From kinetic theory and multiscale methods to applications and research perspectives , 2019, Mathematical Models and Methods in Applied Sciences.

[34]  Seung-Yeal Ha,et al.  Emergent behaviors of the Cucker–Smale ensemble under attractive–repulsive couplings and Rayleigh frictions , 2018, Mathematical Models and Methods in Applied Sciences.

[35]  Lorenzo Pareschi,et al.  Consensus-Based Optimization on the Sphere I: Well-Posedness and Mean-Field Limit , 2020, ArXiv.

[36]  M. Fornasier,et al.  Consensus-based Optimization on the Sphere II: Convergence to Global Minimizers and Machine Learning , 2020, J. Mach. Learn. Res..

[37]  J. Carrillo,et al.  A consensus-based global optimization method for high dimensional machine learning problems , 2019, ESAIM: Control, Optimisation and Calculus of Variations.