Consensus-based Optimization and Ensemble Kalman Inversion for Global Optimization Problems with Constraints

We introduce a practical method for incorporating equality and inequality constraints in global optimization methods based on stochastic interacting particle systems, specifically consensusbased optimization (CBO) and ensemble Kalman inversion (EKI). Unlike other approaches in the literature, the method we propose does not constrain the dynamics to the feasible region of the state space at all times; the particles evolve in the full space, but are attracted towards the feasible set by means of a penalization term added to the objective function and, in the case of CBO, an additional relaxation drift. We study the properties of the method through the associated mean-field Fokker–Planck equation and demonstrate its performance in numerical experiments on several test problems.

[1]  Riccardo Poli,et al.  Particle swarm optimization , 1995, Swarm Intelligence.

[2]  Massimo Fornasier,et al.  Consensus-based optimization methods converge globally in mean-field law , 2021, ArXiv.

[3]  J. Remacle,et al.  Gmsh: A 3‐D finite element mesh generator with built‐in pre‐ and post‐processing facilities , 2009 .

[4]  Andrew Stuart,et al.  Ensemble Kalman methods with constraints , 2019, Inverse problems.

[5]  Claudia Totzeck,et al.  Trends in Consensus-based optimization , 2021, 2104.01383.

[6]  Luca Maria Gambardella,et al.  A survey on metaheuristics for stochastic combinatorial optimization , 2009, Natural Computing.

[7]  Lei Li,et al.  Random Batch Methods (RBM) for interacting particle systems , 2018, J. Comput. Phys..

[8]  Anders Logg,et al.  The FEniCS Project Version 1.5 , 2015 .

[9]  Andrew M. Stuart,et al.  Analysis of the Ensemble Kalman Filter for Inverse Problems , 2016, SIAM J. Numer. Anal..

[10]  Vladas Sidoravicius,et al.  Stochastic Processes and Applications , 2007 .

[11]  Amir Dembo,et al.  Large Deviations Techniques and Applications , 1998 .

[12]  David Márquez.  Convergence rates for annealing diffusion processes , 1997 .

[13]  Nikolas Nüsken,et al.  Affine invariant interacting Langevin dynamics for Bayesian inference , 2020, SIAM J. Appl. Dyn. Syst..

[14]  Sébastien Bubeck,et al.  Convex Optimization: Algorithms and Complexity , 2014, Found. Trends Mach. Learn..

[15]  U. Vaes,et al.  Consensus‐based sampling , 2021, Studies in Applied Mathematics.

[16]  Dervis Karaboga,et al.  A comprehensive survey: artificial bee colony (ABC) algorithm and applications , 2012, Artificial Intelligence Review.

[17]  D. Stroock,et al.  Asymptotics of the spectral gap with applications to the theory of simulated annealing , 1989 .

[18]  Jeffrey Horn,et al.  Handbook of evolutionary computation , 1997 .

[19]  Jinniao Qiu,et al.  On the mean‐field limit for the consensus‐based optimization , 2021, Mathematical Methods in the Applied Sciences.

[20]  D. Stroock,et al.  Simulated annealing via Sobolev inequalities , 1988 .

[21]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[22]  Andrew M. Stuart,et al.  Interacting Langevin Diffusions: Gradient Structure and Ensemble Kalman Sampler , 2019, SIAM J. Appl. Dyn. Syst..

[23]  P. Miller.  Applied asymptotic analysis , 2006 .

[24]  Ehl Emile Aarts,et al.  Simulated annealing and Boltzmann machines , 2003 .

[25]  D. Oliver,et al.  Ensemble Randomized Maximum Likelihood Method as an Iterative Ensemble Smoother , 2011, Mathematical Geosciences.

[26]  R. Pinnau,et al.  A consensus-based model for global optimization and its mean-field limit , 2016, 1604.05648.

[27]  L. Eon Bottou.  Online Learning and Stochastic Approximations , 1998 .

[28]  Andrew M. Stuart,et al.  Inverse problems: A Bayesian perspective , 2010, Acta Numerica.

[29]  Marie-Therese Wolfram,et al.  Ensemble Inference Methods for Models With Noisy and Expensive Likelihoods , 2021, SIAM J. Appl. Dyn. Syst..

[30]  Douglas H. Werner,et al.  The Wind Driven Optimization Technique and its Application in Electromagnetics , 2013, IEEE Transactions on Antennas and Propagation.

[31]  M. Fornasier,et al.  Consensus-based optimization on hypersurfaces: Well-posedness and mean-field limit , 2020, Mathematical Models and Methods in Applied Sciences.

[32]  Jos'e A. Carrillo,et al.  An analytical framework for a consensus-based global optimization method , 2016, 1602.00220.

[33]  Desmond J. Higham,et al.  An Algorithmic Introduction to Numerical Simulation of Stochastic Differential Equations , 2001, SIAM Rev..

[34]  A. Stuart,et al.  Ensemble Kalman methods for inverse problems , 2012, 1209.2736.

[35]  Xin-She Yang,et al.  Firefly Algorithms for Multimodal Optimization , 2009, SAGA.

[36]  M. Pelletier.  Weak convergence rates for stochastic approximation with application to multiple targets and simulated annealing , 1998 .

[37]  Jian-Guo Liu,et al.  Convergence of Random Batch Method for interacting particles with disparate species and weights , 2020, ArXiv.

[38]  Marco Dorigo,et al.  Ant colony optimization theory: A survey , 2005, Theor. Comput. Sci..

[39]  Claudia Schillings,et al.  On the Incorporation of Box-Constraints for Ensemble Kalman Inversion , 2019, Foundations of Data Science.

[40]  Andrew M. Stuart,et al.  Ensemble Kalman inversion: a derivative-free technique for machine learning tasks , 2018, Inverse Problems.

[41]  Albert C. Reynolds,et al.  Investigation of the sampling performance of ensemble-based methods with a simple reservoir model , 2013, Computational Geosciences.

[42]  Deepak V. Kulkarni,et al.  A discontinuous Galerkin formulation for solution of parabolic equations on nonconforming meshes , 2007 .

[43]  Mihai Bostan,et al.  Asymptotic Fixed-Speed Reduced Dynamics for Kinetic Equations in Swarming , 2012, 1202.6557.

[44]  Suvrit Sra,et al.  Fast stochastic optimization on Riemannian manifolds , 2016, ArXiv.

[45]  Shi Jin,et al.  A consensus-based global optimization method for high dimensional machine learning problems , 2019 .

[46]  Chii-Ruey Hwang,et al.  Large-time behavior of perturbed diffusion markov processes with applications to the second eigenvalue problem for fokker-planck operators and simulated annealing , 1990 .

[47]  Xin-She Yang,et al.  A literature survey of benchmark functions for global optimisation problems , 2013, Int. J. Math. Model. Numer. Optimisation.

[48]  Dan Boneh,et al.  On genetic algorithms , 1995, COLT '95.

[49]  U Vaes,et al.  Wasserstein stability estimates for covariance-preconditioned Fokker–Planck equations , 2019, Nonlinearity.

[50]  B. Chandra Mohan,et al.  A survey: Ant Colony Optimization based recent research and implementation on several engineering domain , 2012, Expert Syst. Appl..

[51]  Christian Blum,et al.  Metaheuristics in combinatorial optimization: Overview and conceptual comparison , 2003, CSUR.