On the benefits of Laplace samples in solving a rare event problem using cross-entropy method

The convergence quality of the cross-entropy (CE) optimizer relies critically on the mechanism meant for randomly generating data samples, in agreement with the inference drawn in the earlier works-the fast simulated annealing (FSA) and fast evolutionary programming (FEP). Since tracing a near-global-optimum embedded on a nonconvex search space can be viewed as a rare event problem, a CE algorithm constructed using a long-tailed distribution is intuitively attractive for effectively exploring the optimization landscape. Based on this supposition, a set of CE algorithms employing the Cauchy, logistic and Laplace distributions are experimentally validated in a wide range of optimization functions, which are shifted, rotated, expanded and/or composed, characterized by convex, unimodal, discontinuous, noisy and multimodal fitness landscapes. The Laplace distribution has been demonstrated to be more suitable for the CE optimization, since the samples drawn have jump-lengths long enough to elude local optima and short enough to preserve sufficient candidates in the global optimum neighborhood. Besides, a theoretical analysis has been carried out to understand the following: (i) benefits offered by the long-tailed distributions towards evasion of local optima; (ii) link between the variation in scale parameter estimate and the probability of producing candidate solutions arbitrarily close to the global optimum.

[1]  Dirk P. Kroese,et al.  Convergence properties of the cross-entropy method for discrete optimization , 2007, Oper. Res. Lett..

[2]  Chih-Peng Li,et al.  Cross-Entropy Method for the Optimization of Optical Alignment Signals With Diffractive Effects , 2011, Journal of Lightwave Technology.

[3]  P. Minvielle,et al.  Sparse Antenna Array Optimization With the Cross-Entropy Method , 2011, IEEE Transactions on Antennas and Propagation.

[4]  M. Kendall,et al.  Classical inference and the linear model , 1999 .

[5]  Thomas Bäck,et al.  Evolutionary algorithms in theory and practice - evolution strategies, evolutionary programming, genetic algorithms , 1996 .

[6]  Thomas Bäck,et al.  An Overview of Evolutionary Algorithms for Parameter Optimization , 1993, Evolutionary Computation.

[7]  Kusum Deep,et al.  A new crossover operator for real coded genetic algorithms , 2007, Appl. Math. Comput..

[8]  Ajith Abraham,et al.  New mutation schemes for differential evolution algorithm and their application to the optimization of directional over-current relay settings , 2010, Appl. Math. Comput..

[9]  David B. Fogel,et al.  System Identification Through Simulated Evolution: A Machine Learning Approach to Modeling , 1991 .

[10]  A.M.L. da Silva,et al.  Generating Capacity Reliability Evaluation Based on Monte Carlo Simulation and Cross-Entropy Methods , 2010, IEEE Transactions on Power Systems.

[11]  Jing J. Liang,et al.  Problem Definitions and Evaluation Criteria for the CEC 2005 Special Session on Real-Parameter Optimization , 2005 .

[12]  Liam McDaid,et al.  Landscape classification and problem specific reasoning for genetic algorithms , 2005 .

[13]  Paul A. Viola,et al.  MIMIC: Finding Optima by Estimating Probability Densities , 1996, NIPS.

[14]  H. Szu.  Fast simulated annealing , 1987 .

[15]  J. A. Lozano,et al.  Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation , 2001 .

[16]  Dirk P. Kroese,et al.  Combinatorial Optimization via Cross-Entropy , 2004 .

[17]  Kumar Chellapilla,et al.  Combining mutation operators in evolutionary programming , 1998, IEEE Trans. Evol. Comput..

[18]  Jonathan M. Keith,et al.  Rare event simulation and combinatorial optimization using cross entropy: sequence alignment by rare event simulation , 2002, WSC '02.

[19]  Jin Zhang.  A highly efficient L-estimator for the location parameter of the Cauchy distribution , 2010, Comput. Stat..

[20]  Reuven Y. Rubinstein,et al.  Cross-entropy and rare events for maximal cut and partition problems , 2002, TOMC.

[21]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[22]  Dirk P. Kroese,et al.  Sequence alignment by rare event simulation , 2002, Proceedings of the Winter Simulation Conference.

[23]  Dirk P. Kroese,et al.  The Cross-Entropy Method for Continuous Multi-Extremal Optimization , 2006 .

[24]  Dirk P. Kroese,et al.  Network Reliability Optimization via the Cross-Entropy Method , 2007, IEEE Transactions on Reliability.

[25]  Shie Mannor,et al.  The Cross Entropy Method for Fast Policy Search , 2003, ICML.

[26]  Ad Ridder,et al.  Importance sampling algorithms for first passage time probabilities in the infinite server queue , 2009, Eur. J. Oper. Res..

[27]  Xin Yao,et al.  Evolutionary programming made faster , 1999, IEEE Trans. Evol. Comput..

[28]  Kusum Deep,et al.  Optimal coordination of over-current relays using modified differential evolution algorithms , 2010, Eng. Appl. Artif. Intell..

[29]  Shumeet Baluja,et al.  A Method for Integrating Genetic Search Based Function Optimization and Competitive Learning , 1994 .

[30]  Ajith Abraham,et al.  Differential Evolution with Laplace mutation operator , 2009, 2009 IEEE Congress on Evolutionary Computation.

[31]  Z. D. Bai,et al.  On the maximum‐likelihood estimator for the location parameter of a cauchy distribution , 1987 .

[32]  D. Fogel.  Evolutionary algorithms in theory and practice , 1997, Complex..

[33]  Shiyan Hu,et al.  Hierarchical Cross-Entropy Optimization for Fast On-Chip Decap Budgeting , 2011, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[34]  Avraham Shtub,et al.  Managing Stochastic, Finite Capacity, Multi-Project Systems through the Cross-Entropy Methodology , 2005, Ann. Oper. Res..

[35]  Bart De Schutter,et al.  Cross-Entropy Optimization of Control Policies With Adaptive Basis Functions , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[36]  Reuven Y. Rubinstein.  How Many Needles are in a Haystack, or How to Solve #P-Complete Counting Problems Fast , 2006 .

[37]  Shiyou Yang,et al.  The Cross-Entropy Method and Its Application to Inverse Problems , 2010, IEEE Transactions on Magnetics.

[38]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[39]  Konstantinos G. Margaritis,et al.  On benchmarking functions for genetic algorithms , 2001, Int. J. Comput. Math..

[40]  Kusum Deep,et al.  A new mutation operator for real coded genetic algorithms , 2007, Appl. Math. Comput..

[41]  Hans-Paul Schwefel,et al.  Evolution and Optimum Seeking: The Sixth Generation , 1993 .

[42]  S. Nariai,et al.  On the design of multi-type networks via the cross-entropy method , 2005, DRCN 2005). Proceedings.5th International Workshop on Design of Reliable Communication Networks, 2005..

[43]  Hans-Paul Schwefel,et al.  Evolution and optimum seeking , 1995, Sixth-generation computer technology series.

[44]  Yung-Hwan Oh,et al.  Information Distance-Based Subvector Clustering for ASR Parameter Quantization , 2008, IEEE Signal Processing Letters.