Generality's price: Inescapable deficiencies in machine-learned programs

Abstract This paper investigates some delicate tradeoffs between the generality of an algorithmic learning device and the quality of the programs it learns successfully. There are results to the effect that, thanks to small increases in generality of a learning device, the computational complexity of some successfully learned programs is provably unalterably suboptimal. There are also results in which the complexity of successfully learned programs is asymptotically optimal and the learning device is general, but, still thanks to the generality, some of those optimal, learned programs are provably unalterably information deficient—in some cases, deficient as to safe, algorithmic extractability/provability of the fact that they are even approximately optimal. For these results, the safe, algorithmic methods of information extraction will be by proofs in arbitrary, true, computably axiomatizable extensions of Peano Arithmetic.

[1]  Kenneth W. Regan.  Minimum-Complexity Pairing Functions , 1992, J. Comput. Syst. Sci..

[2]  Jr. Hartley Rogers.  Theory of Recursive Functions and Effective Computability , 1969 .

[3]  Karl Winklmann,et al.  Simple Gödel Numberings, Isomorphisms, and Programming Properties , 1978, SIAM J. Comput..

[4]  James S. Royer,et al.  Subrecursive Programming Systems , 1994, Progress in Theoretical Computer Science.

[5]  Neil D. Jones,et al.  Computability and complexity - from a programming perspective , 1997, Foundations of computing series.

[6]  L. Fortnow.  Counting complexity , 1998 .

[7]  E. Mark Gold,et al.  Language Identification in the Limit , 1967, Inf. Control..

[8]  Clifford Stein,et al.  Introduction to Algorithms, 2nd edition. , 2001 .

[9]  Diana Schmidt.  The Recursion-Theoretic Structure of Complexity Classes , 1985, Theor. Comput. Sci..

[10]  Dexter Kozen,et al.  Indexings of Subrecursive Classes , 1980, Theor. Comput. Sci..

[11]  Richard J. Lipton,et al.  On the Structure of Sets in NP and Other Complexity Classes , 1981, Theor. Comput. Sci..

[12]  John Case,et al.  Comparison of Identification Criteria for Machine Inductive Inference , 1983, Theor. Comput. Sci..

[13]  J. Hartmanis,et al.  On the Computational Complexity of Algorithms , 1965 .

[14]  Richard Edwin Stearns,et al.  Two-Tape Simulation of Multitape Turing Machines , 1966, JACM.

[15]  Richard E. Ladner,et al.  On the Structure of Polynomial Time Reducibility , 1975, JACM.

[16]  Kenneth W. Regan.  The Topology of Provability in Complexity Theory , 1988, J. Comput. Syst. Sci..

[17]  PAUL CHEW,et al.  A Note on Structure and Looking Back Applied to the Complexity of Computable Functions , 1981, J. Comput. Syst. Sci..

[18]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[19]  Manuel Blum,et al.  A Machine-Independent Theory of the Complexity of Recursive Functions , 1967, JACM.

[20]  Robert H. Sloan,et al.  BOOK REVIEW: "SYSTEMS THAT LEARN: AN INTRODUCTION TO LEARNING THEORY, SECOND EDITION", SANJAY JAIN, DANIEL OSHERSON, JAMES S. ROYER and ARUN SHARMA , 2001 .

[21]  John Gill,et al.  Computational Complexity of Probabilistic Turing Machines , 1977, SIAM J. Comput..

[22]  John Case,et al.  Costs of general purpose learning , 2001, Theor. Comput. Sci..

[23]  Umesh V. Vazirani,et al.  Quantum Complexity Theory , 1997, SIAM J. Comput..

[24]  Elliott Mendelson,et al.  Introduction to Mathematical Logic , 1979 .

[25]  Daniel N. Osherson,et al.  Systems That Learn: An Introduction to Learning Theory for Cognitive and Computer Scientists , 1990 .

[26]  S. Kleene.  General recursive functions of natural numbers , 1936 .

[27]  Paul Young,et al.  An introduction to the general theory of algorithms , 1978 .

[28]  Ming Li,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 2019, Texts in Computer Science.

[29]  Keh-Jiann Chen,et al.  Tradeoffs in machine inductive inference , 1981 .

[30]  Uwe Schöning.  A Uniform Approach to Obtain Diagonal Sets in Complexity Classes , 1982, Theor. Comput. Sci..

[31]  Ronald L. Rivest,et al.  Introduction to Algorithms , 1990 .

[32]  A. Selman,et al.  Complexity theory retrospective II , 1998 .

[33]  Manuel Blum,et al.  Toward a Mathematical Theory of Inductive Inference , 1975, Inf. Control..

[34]  J. Case,et al.  Subrecursive Programming Systems: Complexity & Succinctness , 1994 .