Survey on computational complexity with phase transitions and extremal optimization

Applying statistical mechanics to search problems in AI, decisions and optimization has been one of the powerful channels to solve NP-hard problems. Extensive analytical and experimental research has shown that the “phase transition” phenomenon in search space is often associated with the hardness of complexity. A Bak-Sneppen (BS) model based general-purpose heuristic method, called extremal optimization (EO), proposed by Boettcher and Percus from physics society may perform very well, especially near the phase transitions in compared with other optimization methods, e.g., genetic algorithm and simulated annealing, etc. To actuate more extensive investigations on this new optimization approach particularly in control, computer and optimization communities, this survey reviews the latest research results from fundamental to practice about the connection between computational complexity and phase transitions. Then, further introduces the concepts, fundamentals, algorithms and applications of EO from its capability of self-organized criticality, backbone analysis and co-evolution moving to a far-from-equilibrium state. Finally, the concluding remarks with suggested future research are illustrated.

[1]  G. Unter Rudolph.  Convergence Rates of Evolutionary Algorithms for a Class of Convex Objective Functions , 1997 .

[2]  Joseph C. Culberson,et al.  Frozen development in graph coloring , 2001, Theor. Comput. Sci..

[3]  Joe Suzuki,et al.  A Markov chain analysis on simple genetic algorithms , 1995, IEEE Trans. Syst. Man Cybern..

[4]  Chrystopher L. Nehaniv,et al.  Linear Analysis of Genetic Algorithms , 1998, Theor. Comput. Sci..

[5]  D. Garlaschelli,et al.  Self-organized network evolution coupled to extremal dynamics , 2006, cond-mat/0611201.

[6]  R. Tweedie,et al.  Rates of convergence of the Hastings and Metropolis algorithms , 1996 .

[7]  J. Rosenthal.  Minorization Conditions and Convergence Rates for Markov Chain Monte Carlo , 1995 .

[8]  Yang Genke,et al.  Multiobjective extremal optimization with applications to engineering design , 2007 .

[9]  José Carlos Príncipe,et al.  A Markov Chain Framework for the Simple Genetic Algorithm , 1993, Evolutionary Computation.

[10]  Peter Reynolds,et al.  Large-cell renormalisation group for the backbone problem in percolation , 1979 .

[11]  Toby Walsh,et al.  The TSP Phase Transition , 1996, Artif. Intell..

[12]  H. Jane Brockmann,et al.  The selfish gene (2nd edn) , 1990 .

[13]  Weixiong Zhang,et al.  A Study of Complexity Transitions on the Asymmetric Traveling Salesman Problem , 1996, Artif. Intell..

[14]  A. Arenas,et al.  Community detection in complex networks using extremal optimization. , 2005, Physical review. E, Statistical, nonlinear, and soft matter physics.

[15]  Yu-Wang Chen,et al.  Gene Optimization: Computational Intelligence from the Natures and Micro-mechanisms of Hard Computational Systems , 2007, LSMS.

[16]  Wei Li,et al.  Many hard examples in exact phase transitions , 2003, Theor. Comput. Sci..

[17]  Dan Boneh,et al.  On genetic algorithms , 1995, COLT '95.

[18]  Mohamed Batouche,et al.  Robust Point Correspondence for Image Registration Using Optimization with Extremal Dynamics , 2002, DAGM-Symposium.

[19]  Bart Selman,et al.  Generating Satisfiable Problem Instances , 2000, AAAI/IAAI.

[20]  Ricard V. Solé,et al.  Phase Transitions in a Model of Internet Traffic , 2000 .

[21]  Gunar E. Liepins,et al.  Punctuated Equilibria in Genetic Search , 1991, Complex Syst..

[22]  Goldberg,et al.  Genetic algorithms , 1993, Robust Control Systems with Genetic Algorithms.

[23]  Toby Walsh,et al.  Backbones in Optimization and Approximation , 2001, IJCAI.

[24]  Michael D. Vose,et al.  Random Heuristic Search , 1999, Theor. Comput. Sci..

[25]  dos Astronautas.  Function Optimization Using Extremal Dynamics , 2002 .

[26]  Mathieu S. Capcarrère,et al.  Necessary conditions for density classification by cellular automata. , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[27]  Joe Suzuki,et al.  A Further Result on the Markov Chain Model of Genetic Algorithms and Its Application to a Simulated Annealing-like Strategy , 1998, FOGA.

[28]  Stefan Boettcher,et al.  Extremal Optimization for the Sherrington-Kirkpatrick Spin Glass , 2004 .

[29]  Stefan Boettcher.  Extremal Optimization: Heuristics Via Co-Evolutionary Avalanches , 2000, Comput. Sci. Eng..

[30]  J. C. Jackson.  Constrainedness and the Phase Transition in Job Shop Scheduling , 1997 .

[31]  Faster Monte Carlo simulations at low temperatures. The waiting time method , 2001, cond-mat/0107475.

[32]  R. Cerf.  Asymptotic convergence of genetic algorithms , 1998, Advances in Applied Probability.

[33]  Stephan Mertens,et al.  Phase transition in multiprocessor scheduling. , 2003, Physical review letters.

[34]  David E. Goldberg,et al.  Finite Markov Chain Analysis of Genetic Algorithms , 1987, ICGA.

[35]  Weixiong Zhang,et al.  Phase Transitions and Backbones of the Asymmetric Traveling Salesman Problem , 2011, J. Artif. Intell. Res..

[36]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[37]  Stefan Boettcher,et al.  Extremal Optimization for Graph Partitioning , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[38]  Andrew J. Parkes,et al.  Clustering at the Phase Transition , 1997, AAAI/IAAI.

[39]  S Kirkpatrick,et al.  Critical Behavior in the Satisfiability of Random Boolean Expressions , 1994, Science.

[40]  Benton,et al.  Criticality and scaling in evolutionary ecology. , 1997, Trends in ecology & evolution.

[41]  Min-Rong Chen,et al.  Population-Based Extremal Optimization with Adaptive Lévy Mutation for Constrained Optimization , 2006, 2006 International Conference on Computational Intelligence and Security.

[42]  Bert De Reyck,et al.  DEPARTEMENT TOEGEPASTE ECONOMISCHE WETENSCHAPPEN PHASE TRANSITIONS IN PROJECT SCHEDULING , 2022 .

[43]  P. Anderson,et al.  Application of statistical mechanics to NP-complete problems in combinatorial optimisation , 1986 .

[44]  Mohamed El Bachir Menai,et al.  A Bose-Einstein Extremal Optimization Method for Solving Real-World Instances of Maximum Satisfiablility , 2003, IC-AI.

[45]  Bart Selman,et al.  Computational science: A hard statistical view , 2008, Nature.

[46]  Tang,et al.  Self-Organized Criticality: An Explanation of 1/f Noise , 2011 .

[47]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[48]  Stefan Boettcher,et al.  Extremal Optimization: an Evolutionary Local-Search Algorithm , 2002, ArXiv.

[49]  Fabiano Luis de Sousa,et al.  Generalized extremal optimization: An application in heat pipe design , 2004 .

[50]  Pedro Paglione,et al.  New Stochastic Algorithm for Design Optimization , 2003 .

[51]  Lishan Kang,et al.  On the Convergence Rates of Genetic Algorithms , 1999, Theor. Comput. Sci..

[52]  Mohamed Batouche,et al.  Combining Extremal Optimization With Singular Value Decomposition For Effective Point Matching , 2003, Int. J. Pattern Recognit. Artif. Intell..

[53]  Rémi Monasson,et al.  Determining computational complexity from characteristic ‘phase transitions’ , 1999, Nature.

[54]  A. Middleton,et al.  Improved extremal optimization for the Ising spin glass. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[55]  Mohamed Batouche,et al.  Ant colony system with extremal dynamics for point matching and pose estimation , 2002, Object recognition supported by user interaction for service robots.

[56]  Heidelberg,et al.  Encircling an exceptional point. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[57]  Stefan Boettcher,et al.  Optimizing at the ergodic edge , 2006 .

[58]  Min-Rong Chen,et al.  A novel elitist multiobjective optimization algorithm: Multiobjective extremal optimization , 2008, Eur. J. Oper. Res..

[59]  Richard L. Tweedie,et al.  Geometric Convergence Rates for Stochastically Ordered Markov Chains , 1996, Math. Oper. Res..

[60]  Karl Heinz Hoffmann,et al.  Fitness threshold accepting over extremal optimization ranks. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[61]  Stefan Boettcher,et al.  Extremal Optimization: Methods derived from Co-Evolution , 1999, GECCO.

[62]  Philip W. Anderson.  Solving problems in finite time: Computing , 1999 .

[63]  Mohamed El Bachir Menai,et al.  A Backbone-Based Co-evolutionary Heuristic for Partial MAX-SAT , 2005, Artificial Evolution.

[64]  Kay Hamacher,et al.  Adaptive extremal optimization by detrended fluctuation analysis , 2007, J. Comput. Phys..

[65]  Stephen A. Cook,et al.  The complexity of theorem-proving procedures , 1971, STOC.

[66]  J. M. Singer,et al.  Searching for backbones — an efficient parallel algorithm for the traveling salesman problem , 1996 .

[67]  Tang,et al.  Self-organized criticality. , 1988, Physical review. A, General physics.

[68]  Weixiong Zhang,et al.  Configuration landscape analysis and backbone guided local search: Part I: Satisfiability and maximum satisfiability , 2004, Artif. Intell..

[69]  Peng Chen,et al.  Optimization with extremal dynamics for the traveling salesman problem , 2007 .

[70]  Andrea Montanari,et al.  Gibbs states and the set of solutions of random constraint satisfaction problems , 2006, Proceedings of the National Academy of Sciences.

[71]  Tad Hogg,et al.  Phase Transitions and the Search Problem , 1996, Artif. Intell..

[72]  Stefan Boettcher,et al.  Optimization with extremal dynamics , 2003, Complex..

[73]  Tao Zhou,et al.  Continuous extremal optimization for Lennard-Jones clusters. , 2005, Physical review. E, Statistical, nonlinear, and soft matter physics.

[74]  Lothar M. Schmitt,et al.  Theory of genetic algorithms , 2001, Theor. Comput. Sci..

[75]  Mohamed El Bachir Menai,et al.  Efficient Initial Solution to Extremal Optimization Algorithm for Weighted MAXSAT Problem , 2003, IEA/AIE.

[76]  Bak,et al.  Punctuated equilibrium and criticality in a simple model of evolution. , 1993, Physical review letters.

[77]  Assaf Naor,et al.  Rigorous location of phase transitions in hard optimization problems , 2005, Nature.

[78]  Kalyanmoy Deb,et al.  Messy Genetic Algorithms: Motivation, Analysis, and First Results , 1989, Complex Syst..

[79]  L. Darrell Whitley,et al.  Problem difficulty for tabu search in job-shop scheduling , 2003, Artif. Intell..

[80]  S. Boettcher.  Extremal optimization for Sherrington-Kirkpatrick spin glasses , 2004, cond-mat/0407130.

[81]  Peter C. Cheeseman,et al.  Where the Really Hard Problems Are , 1991, IJCAI.

[82]  Stefan Boettcher,et al.  Comparing extremal and thermal explorations of energy landscapes , 2005 .

[83]  Rémi Monasson,et al.  Statistical mechanics methods and phase transitions in optimization problems , 2001, Theor. Comput. Sci..

[84]  Alden H. Wright,et al.  Simple Genetic Algorithms with Linear Fitness , 1994, Evolutionary Computation.

[85]  Athanassios Siapas,et al.  Criticality and Parallelism in Combinatorial Optimization , 1996, Science.

[86]  R. Zecchina,et al.  Phase transitions in combinatorial problems , 2001 .

[87]  Fabiano Luis de Sousa,et al.  Generalized Extremal Optimization for Solving Complex Optimal Design Problems , 2003, GECCO.

[88]  M. Mézard,et al.  Analytic and Algorithmic Solution of Random Satisfiability Problems , 2002, Science.

[89]  Gareth O. Roberts,et al.  Quantitative Bounds for Convergence Rates of Continuous Time Markov Processes , 1996 .

[90]  Fabiano Luis de Sousa,et al.  Heat Pipe Design Through Generalized Extremal Optimization , 2004 .

[91]  B. Selman,et al.  Satisfied with Physics , 2002, Science.

[92]  Stefan Boettcher,et al.  Optimization with Extremal Dynamics , 2000, Complex..

[93]  Weixiong Zhang,et al.  A Novel Local Search Algorithm for the Traveling Salesman Problem that Exploits Backbones , 2005, IJCAI.

[94]  Florent Krzakala,et al.  A Landscape Analysis of Constraint Satisfaction Problems , 2007, Physical review. E, Statistical, nonlinear, and soft matter physics.

[95]  Jose C. Principe,et al.  A Markov Framework for the Simple Genetic Algorithm , 1998 .

[96]  O. Martin,et al.  The Stochastic Traveling Salesman Problem: Finite Size Scaling and the Cavity Prediction , 1998, cond-mat/9802295.

[97]  Yu-Wang Chen,et al.  Hybrid evolutionary algorithm with marriage of genetic algorithm and extremal optimization for production scheduling , 2008 .

[98]  Stefan Boettcher,et al.  Spines of random constraint satisfaction problems: definition and connection with computational complexity , 2005, Annals of Mathematics and Artificial Intelligence.

[99]  Rémi Monasson,et al.  A review of the Statistical Mechanics approach to Random Optimization Problems , 2008, ArXiv.

[100]  Min-Rong Chen,et al.  Multiobjective optimization using population-based extremal optimization , 2008, Neural Computing and Applications.

[101]  R. N. Onody,et al.  Optimization and self-organized criticality in a magnetic system , 2002 .

[102]  Johannes Schneider,et al.  Searching for Backbones--a high-performance parallel algorithm for solving combinatorial optimization problems , 2003, Future Gener. Comput. Syst..

[103]  Bart Selman,et al.  Computational science: Can get satisfaction , 2005, Nature.

[104]  Stefan Boettcher,et al.  Extremal optimization at the phase transition of the three-coloring problem. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[105]  Stefan Boettcher,et al.  Jamming Model for the Extremal Optimization Heuristic , 2001, ArXiv.

[106]  Min-Rong Chen,et al.  Studies on Extremal Optimization and Its Applications in Solving RealWorld Optimization Problems , 2007, 2007 IEEE Symposium on Foundations of Computational Intelligence.

[107]  Lothar M. Schmitt,et al.  Theory of Genetic Algorithms II: models for genetic operators over the string-tensor representation of populations and convergence to global optima for arbitrary fitness function under scaling , 2004, Theor. Comput. Sci..

[108]  Günter Rudolph,et al.  Convergence analysis of canonical genetic algorithms , 1994, IEEE Trans. Neural Networks.

[109]  A. Hartmann.  Phase Transitions in Combinatorial Optimization Problems - Basics, Algorithms and Statistical Mechanics , 2005 .

[110]  Karl Heinz Hoffmann,et al.  Best possible probability distribution over extremal optimization ranks , 2004 .

[111]  Yutaka Okabe,et al.  A Comparison of Extremal Optimization with Flat-Histogram and Equal-Hit Dynamics for Finding Spin–Glass Ground States , 2002 .

[112]  Evangelos Kranakis,et al.  Special Issue on Typical Case Complexity and Phase Transitions , 2005, Discret. Appl. Math..

[113]  Alan Smaill,et al.  Backbone Fragility and the Local Search Cost Peak , 2000, J. Artif. Intell. Res..

[114]  Gary J. Koehler,et al.  A Markov chain analysis of genetic algorithms with power of 2 cardinality alphabets , 1997 .

[115]  S. Boettcher.  Extremal Optimization of Graph Partitioning at the Percolation Threshold , 1999, cond-mat/9901353.

[116]  Elad Yom-Tov,et al.  Movement-related potentials during the performance of a motor task I: The effect of learning and force , 2001, Biological Cybernetics.

[117]  Wei Li,et al.  The SAT phase transition , 1999, ArXiv.

[118]  A. Percus,et al.  Nature's Way of Optimizing , 1999, Artif. Intell..

[119]  Weixiong Zhang,et al.  Phase Transitions and Backbones of 3-SAT and Maximum 3-SAT , 2001, CP.

[120]  M. Mézard.  Passing Messages Between Disciplines , 2003, Science.