Generality's Price: Inescapable Deficiencies in Machine-Learned Programs

This paper investigates some delicate tradeoffs between the generality of an algorithmic learning device and the quality of the programs it learns successfully. There are results to the effect that, thanks to small increases in generality of a learning device, the computational complexity of some successfully learned programs is provably unalterably suboptimal. There are also results in which the complexity of successfully learned programs is asymptotically optimal and the learning device is general, but, still thanks to the generality, some of those optimal, learned programs are provably unalterably information deficient -in some cases, deficient as to safe, algorithmic extractability/provability of the fact that they are even approximately optimal. The paper is on the borderline between learning theory, complexity theory and logic.

[1]  James S. Royer,et al.  Subrecursive Programming Systems , 1994, Progress in Theoretical Computer Science.

[2]  Neil D. Jones,et al.  Computability and complexity - from a programming perspective , 1997, Foundations of computing series.

[3]  William I. Gasarch,et al.  Book Review: An introduction to Kolmogorov Complexity and its Applications Second Edition, 1997 by Ming Li and Paul Vitanyi (Springer (Graduate Text Series)) , 1997, SIGACT News.

[4]  Keh-Jiann Chen,et al.  Tradeoffs in machine inductive inference , 1981 .

[5]  D. C. Cooper,et al.  Theory of Recursive Functions and Effective Computability , 1969, The Mathematical Gazette.

[6]  L. Fortnow.  Counting complexity , 1998 .

[7]  Paul Young,et al.  An introduction to the general theory of algorithms , 1978 .

[8]  Uwe Schöning.  A Uniform Approach to Obtain Diagonal Sets in Complexity Classes , 1982, Theor. Comput. Sci..

[9]  Dexter Kozen,et al.  Indexings of Subrecursive Classes , 1980, Theor. Comput. Sci..

[10]  Richard J. Lipton,et al.  On the Structure of Sets in NP and Other Complexity Classes , 1981, Theor. Comput. Sci..

[11]  Hans Hermes,et al.  Introduction to mathematical logic , 1973, Universitext.

[12]  J. Hartmanis,et al.  On the Computational Complexity of Algorithms , 1965 .

[13]  Manuel Blum,et al.  Toward a Mathematical Theory of Inductive Inference , 1975, Inf. Control..

[14]  J. Case,et al.  Subrecursive Programming Systems: Complexity & Succinctness , 1994 .

[15]  Richard Edwin Stearns,et al.  Two-Tape Simulation of Multitape Turing Machines , 1966, JACM.

[16]  Paul M. B. Vitányi,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 1997, Graduate Texts in Computer Science.

[17]  Kenneth W. Regan.  The Topology of Provability in Complexity Theory , 1986, Computational Complexity Conference.

[18]  Kenneth W. Regan.  Minimum-Complexity Pairing Functions , 1992, J. Comput. Syst. Sci..

[19]  Daniel N. Osherson,et al.  Systems That Learn: An Introduction to Learning Theory for Cognitive and Computer Scientists , 1990 .

[20]  John Case,et al.  Costs of General Purpose Learning , 1999, STACS.

[21]  Diana Schmidt.  The Recursion-Theoretic Structure of Complexity Classes , 1985, Theor. Comput. Sci..

[22]  Jr. Hartley Rogers.  Theory of Recursive Functions and Effective Computability , 1969 .

[23]  Umesh V. Vazirani,et al.  Quantum Complexity Theory , 1997, SIAM J. Comput..

[24]  E. Mark Gold,et al.  Language Identification in the Limit , 1967, Inf. Control..

[25]  John Case,et al.  Comparison of Identification Criteria for Machine Inductive Inference , 1983, Theor. Comput. Sci..

[26]  Manuel Blum,et al.  A Machine-Independent Theory of the Complexity of Recursive Functions , 1967, JACM.

[27]  Robert H. Sloan,et al.  BOOK REVIEW: "SYSTEMS THAT LEARN: AN INTRODUCTION TO LEARNING THEORY, SECOND EDITION", SANJAY JAIN, DANIEL OSHERSON, JAMES S. ROYER and ARUN SHARMA , 2001 .

[28]  John Case,et al.  Costs of general purpose learning , 2001, Theor. Comput. Sci..

[29]  S. Kleene.  General recursive functions of natural numbers , 1936 .

[30]  Richard E. Ladner,et al.  On the Structure of Polynomial Time Reducibility , 1975, JACM.

[31]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .