Bridging Supervised Learning and Test-Based Co-optimization

This paper takes a close look at the important commonalities and subtle differences between the well-established field of supervised learning and the much younger one of cooptimization. It explains the relationships between the problems, algorithms and views on cost and performance of the two fields, all throughout providing a two-way dictionary for the respective terminologies used to describe these concepts. The intent is to facilitate advancement of both fields through transfer and cross-pollination of ideas, techniques and results. As a proof of concept, a theoretical study is presented on the connection between existence / lack of free lunch in the two fields, showcasing a few ideas for improving computational complexity of certain supervised learning approaches.

[1]  Elena Popovici,et al.  On the practicality of optimal output mechanisms for co-optimization algorithms , 2011, FOGA '11.

[2]  M. Stone.  Cross‐Validatory Choice and Assessment of Statistical Predictions , 1976 .

[3]  Peter J. Angeline,et al.  Competitive Environments Evolve Better Solutions for Complex Tasks , 1993, ICGA.

[4]  W. Daniel Hillis,et al.  Co-evolving parasites improve simulated evolution as an optimization procedure , 1990 .

[5]  Elena Popovici,et al.  An analysis of two-population coevolutionary computation , 2006 .

[6]  Thomas Reinartz,et al.  A Unifying View on Instance Selection , 2002, Data Mining and Knowledge Discovery.

[7]  Thomas J. Santner,et al.  The Design and Analysis of Computer Experiments , 2003, Springer Series in Statistics.

[8]  Elena Popovici,et al.  A framework for co-optimization algorithm performance and its application to worst-case optimization , 2015, Theor. Comput. Sci..

[9]  Nikolaus Hansen,et al.  Completely Derandomized Self-Adaptation in Evolution Strategies , 2001, Evolutionary Computation.

[10]  Edwin D. de Jong.  Objective fitness correlation , 2007, GECCO '07.

[11]  Burr Settles,et al.  Active Learning , 2012, Synthesis Lectures on Artificial Intelligence and Machine Learning.

[12]  Elena Popovici.  Co-Optimization Free Lunches: Tractability of Optimal Black-Box Algorithms for Maximizing Expected Utility , 2018, Evolutionary Computation.

[13]  Stewart W. Wilson,et al.  Learning Classifier Systems, From Foundations to Applications , 2000 .

[14]  Toniann Pitassi,et al.  The reusable holdout: Preserving validity in adaptive data analysis , 2015, Science.

[15]  Josh C. Bongard,et al.  Improving genetic programming based symbolic regression using deterministic machine learning , 2013, 2013 IEEE Congress on Evolutionary Computation.

[16]  Daniel R. Tauritz,et al.  A no-free-lunch framework for coevolution , 2008, GECCO '08.

[17]  Hsuan-Tien Lin,et al.  Learning From Data , 2012 .

[18]  Daniel R. Tauritz,et al.  Co-optimization algorithms , 2008, GECCO '08.

[19]  Andreas Holzinger,et al.  Data Mining with Decision Trees: Theory and Applications , 2015, Online Inf. Rev..

[20]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[21]  Elie Bienenstock,et al.  Neural Networks and the Bias/Variance Dilemma , 1992, Neural Computation.

[22]  安藤 寛,et al.  Cross-Validation , 1952, Encyclopedia of Machine Learning and Data Mining.

[23]  Thomas J. Santner,et al.  Design and analysis of computer experiments , 1998 .

[24]  Seymour Geisser,et al.  The Predictive Sample Reuse Method with Applications , 1975 .

[25]  Jordan B. Pollack,et al.  Emergent geometric organization and informative dimensions in coevolutionary algorithms , 2007 .

[26]  Subhransu Maji.  Large Scale Image Annotations on Amazon Mechanical Turk , 2011 .

[27]  Jianlin Cheng,et al.  Machine Learning Methods for Protein Structure Prediction , 2008, IEEE Reviews in Biomedical Engineering.

[28]  Richard K. Belew,et al.  New Methods for Competitive Coevolution , 1997, Evolutionary Computation.

[29]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[30]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[31]  David H. Wolpert,et al.  On the Connection between In-sample Testing and Generalization Error , 1992, Complex Syst..

[32]  Edwin D. de Jong,et al.  Automated Extraction of Problem Structure , 2004, GECCO.

[33]  Dario Floreano,et al.  Neuroevolution: from architectures to learning , 2008, Evol. Intell..

[34]  Sean Luke,et al.  A Comparison Of Two Competitive Fitness Functions , 2002, GECCO.

[35]  Peter Tiño,et al.  Measuring Generalization Performance in Coevolutionary Learning , 2008, IEEE Transactions on Evolutionary Computation.

[36]  David H. Wolpert,et al.  The Lack of A Priori Distinctions Between Learning Algorithms , 1996, Neural Computation.

[37]  Daniel R. Tauritz,et al.  Free lunches in pareto coevolution , 2009, GECCO '09.

[38]  Edwin D. de Jong,et al.  The MaxSolve algorithm for coevolution , 2005, GECCO '05.

[39]  David H. Wolpert,et al.  Coevolutionary free lunches , 2005, IEEE Transactions on Evolutionary Computation.

[40]  Shlomo Argamon,et al.  Committee-Based Sampling For Training Probabilistic Classi(cid:12)ers , 1995 .

[41]  Travis C. Service.  Unbiased coevolutionary solution concepts , 2009, FOGA.

[42]  Kent A. Spackman,et al.  Signal Detection Theory: Valuable Tools for Evaluating Inductive Learning , 1989, ML.

[43]  Mikkel T. Jensen.  Finding worst-case flexible schedules using coevolution , 2001 .

[44]  Edwin D. de Jong,et al.  Coevolutionary Principles , 2012, Handbook of Natural Computing.

[45]  Kenneth A. De Jong,et al.  Relationships between internal and external metrics in co-evolution , 2005, 2005 IEEE Congress on Evolutionary Computation.

[46]  John R. Koza,et al.  Genetic Programming IV: Routine Human-Competitive Machine Intelligence , 2003 .

[47]  J. A. Lozano,et al.  Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation , 2001 .

[48]  Kenneth A. De Jong,et al.  Monotonicity versus performance in co-optimization , 2009, FOGA '09.

[49]  Mitchell A. Potter,et al.  The design and analysis of a computational model of cooperative coevolution , 1997 .

[50]  Kenneth A. De Jong.  Evolutionary Computation: A Unified Approach , 2015, GECCO.