Parsimony hierarchies for inductive inference

Freivalds defined an acceptable programming system independent criterion for learning programs for functions in which the final programs were required to be both correct and “nearly” minimal size, i.e, within a computable function of being purely minimal size. Kinber showed that this parsimony requirement on final programs limits learning power. However, in scientific inference, parsimony is considered highly desirable. A lim-computable function is (by definition) one calculable by a total procedure allowed to change its mind finitely many times about its output. Investigated is the possibility of assuaging somewhat the limitation on learning power resulting from requiring parsimonious final programs by use of criteria which require the final, correct programs to be “not-so-nearly” minimal size, e.g., to be within a lim-computable function of actual minimal size. It is shown that some parsimony in the final program is thereby retained, yet learning power strictly increases. Considered, then, are lim-computable functions as above but for which notations for constructive ordinals are used to bound the number of mind changes allowed regarding the output. This is a variant of an idea introduced by Freivalds and Smith. For this ordinal notation complexity bounded version of lim-computability, the power of the resultant learning criteria form finely graded, infinitely ramifying, infinite hierarchies intermediate between the computable and the lim-computable cases. Some of these hierarchies, for the natural notations determining them, are shown to be optimally tight.

[1]  James S. Royer A Connotational Theory of Program Structure , 1987, Lecture Notes in Computer Science.

[2]  R. Harré,et al.  Probability and Confirmation , 1970 .

[3]  Paul Young,et al.  An introduction to the general theory of algorithms , 1978 .

[4]  Rusins Frievalds Inductive inference of minimal programs , 1990, COLT '90.

[5]  Arun Sharma,et al.  Mind change complexity of learning logic programs , 1999, Theor. Comput. Sci..

[6]  Dana Angluin,et al.  Finding Patterns Common to a Set of Strings , 1980, J. Comput. Syst. Sci..

[7]  C. Dilworth Probability and Confirmation , 1988 .

[8]  John B. Shoven,et al.  I , Edinburgh Medical and Surgical Journal.

[9]  Ivan Bratko,et al.  Applications of inductive logic programming , 1995, SGAR.

[10]  Rolf Wiehagen,et al.  Inductive Inference with Additional Information , 1979, J. Inf. Process. Cybern..

[11]  S. Kleene On the Forms of the Predicates in the Theory of Constructive Ordinals (Second Paper) , 1955 .

[12]  Y. Ershov On a hierarchy of sets, II , 1968 .

[13]  Daniel N. Osherson,et al.  Systems That Learn: An Introduction to Learning Theory for Cognitive and Computer Scientists , 1990 .

[14]  John Case,et al.  Comparison of Identification Criteria for Machine Inductive Inference , 1983, Theor. Comput. Sci..

[15]  Ivan Bratko,et al.  Applications of inductive logic programming , 1995, CACM.

[16]  Kalvis Apsitis Derived Sets and Inductive Inference , 1994, AII/ALT.

[17]  Stephen Cole Kleene,et al.  On notation for ordinal numbers , 1938, Journal of Symbolic Logic.

[18]  Raymond M. Smullyan,et al.  Theory of Formal Systems. (AM-47) , 1961 .

[19]  Robert Nix,et al.  Editing by example , 1985, POPL '84.

[20]  Carl H. Smith,et al.  Inductive Inference with Procrastination: Back to Definitions , 1999, Fundam. Informaticae.

[21]  Esko Ukkonen,et al.  Discovering Unbounded Unions of Regular Pattern Languages from Positive Examples (Extended Abstract) , 1996, ISAAC.

[22]  Setsuo Arikawa,et al.  Pattern Inference , 1995, GOSLER Final Report.

[23]  Saso Dzeroski,et al.  Inductive Logic Programming: Techniques and Applications , 1993 .

[24]  Andris Ambainis,et al.  Ordinal Mind Change Complexity of Language Identification , 1997, Theor. Comput. Sci..

[25]  Luc De Raedt,et al.  Inductive Logic Programming: Theory and Methods , 1994, J. Log. Program..

[26]  Manuel Blum,et al.  Toward a Mathematical Theory of Inductive Inference , 1975, Inf. Control..

[27]  P. Odifreddi Classical recursion theory , 1989 .

[28]  Ayumi Shinohara,et al.  Knowledge Acquisition from Amino Acid Sequences by Machine Learning System BONSAI , 1992 .

[29]  Keith Wright Identification of unions of languages drawn from an identifiable class , 1989, COLT '89.

[30]  N. Shapiro Review: E. Mark Gold, Limiting Recursion; Hilary Putnam, Trial and Error Predicates and the Solution to a Problem of Mostowski , 1971 .

[31]  R. Epstein,et al.  Hierarchies of sets and degrees below 0 , 1981 .

[32]  Y. Ershov A hierarchy of sets. I , 1968 .

[33]  Yu. L. Ershov,et al.  On a hierarchy of sets. III , 1968 .

[34]  Rusins Freivalds Inductive Inference of Minimal Programs , 1990, COLT.

[35]  Carl H. Smith,et al.  Recursion theoretic models of learning: Some results and intuitions , 2005, Annals of Mathematics and Artificial Intelligence.

[36]  Rodney G. Downey,et al.  On Genericity and Ershov's Hierarchy , 2001, Math. Log. Q..

[37]  John Case,et al.  Predictive Learning Models for Concept Drift , 1998, ALT.

[38]  Keh-Jiann Chen Tradeoffs in the Inductive Inference of Nearly Minimal Size Programs , 1982, Inf. Control..

[39]  Thomas G. Dietterich What is machine learning? , 2020, Archives of Disease in Childhood.

[40]  John Case,et al.  Predictive learning models for concept drift , 2001, Theor. Comput. Sci..

[41]  Robert I. Soare,et al.  Computability and Recursion , 1996, Bulletin of Symbolic Logic.

[42]  Marcus Schaefer A guided tour of minimal indices and shortest descriptions , 1998, Arch. Math. Log..

[43]  Hartley Rogers,et al.  Gödel numberings of partial recursive functions , 1958, Journal of Symbolic Logic.

[44]  Rózsa Péter Review: S. C. Kleene, On Notation for Ordinal Numbers , 1939 .

[45]  S. Kleene On the Forms of the Predicates in the Theory of Constructive Ordinals , 1944 .

[46]  Arun Sharma,et al.  Generalized notions of mind change complexity , 1997, COLT '97.

[47]  Jr. Hartley Rogers Theory of Recursive Functions and Effective Computability , 1969 .

[48]  Heikki Mannila,et al.  MDL learning of unions of simple pattern languages from positive examples , 1995, EuroCOLT.

[49]  Arun Sharma,et al.  Elementary formal systems, intrinsic complexity, and procrastination , 1997, COLT '96.

[50]  Manuel Blum,et al.  A Machine-Independent Theory of the Complexity of Recursive Functions , 1967, JACM.

[51]  Andris Ambainis The power of procrastination in inductive inference: How it depends on used ordinal notations , 1995, EuroCOLT.

[52]  John Case,et al.  Machine Induction Without Revolutionary Changes in Hypothesis Size , 1996, Inf. Comput..

[53]  Carl H. Smith,et al.  On the Role of Procrastination in Machine Learning , 1993, Inf. Comput..

[54]  Arto Salomaa,et al.  Return to Patterns , 1995, Bull. EATCS.

[55]  John Case,et al.  Infinitary self-reference in learning theory , 1994, J. Exp. Theor. Artif. Intell..

[56]  E. Mark Gold,et al.  Language Identification in the Limit , 1967, Inf. Control..

[57]  Akihiro Yamamoto,et al.  Learning Elementary Formal Systems , 1992, Theor. Comput. Sci..

[58]  V. L. Selivanov Hierarchy of limiting computations , 1984 .

[59]  Hilary Putnam,et al.  Trial and error predicates and the solution to a problem of Mostowski , 1965, Journal of Symbolic Logic.

[60]  John Case,et al.  Periodicity in generations of automata , 1974, Mathematical systems theory.

[61]  R. V. Freivald Minimal Gödel Numbers and Their Identification in the Limit , 1975, MFCS.

[62]  T. Shinohara INFERRING UNIONS OF TWO PATTERN LANGUAGES , 1983 .

[63]  G. Sacks Higher recursion theory , 1990 .

[64]  Steffen Lange,et al.  Algorithmic Learning for Knowledge-Based Systems , 1995, Lecture Notes in Computer Science.

[65]  Wacław Sierpiński,et al.  Cardinal and Ordinal Numbers , 1966 .

[66]  John Case,et al.  Incremental Concept Learning for Bounded Data Mining , 1997, Inf. Comput..

[67]  Mark A. Fulk A study of inductive inference machines , 1986 .

[68]  Arun Sharma,et al.  On a generalized notion of mistake bounds , 1999, COLT '99.

[69]  John Case,et al.  Not-So-Nearly-Minimal-Size Program Inference , 1995, GOSLER Final Report.

[70]  Carl H. Smith,et al.  On the role of procrastination for machine learning , 1992, COLT '92.

[71]  R. Smullyan Theory of formal systems , 1962 .