Simplicity and likelihood: An axiomatic approach

We suggest a model in which theories are ranked given various databases. Certain axioms on such rankings imply a numerical representation that is the sum of the log-likelihood of the theory and a fixed number for each theory, which may be interpreted as a measure of its complexity. This additive combination of log-likelihood and a measure of complexity generalizes both the Akaike Information Criterion and the Minimum Description Length criterion, which are well known in statistics and in machine learning, respectively. The axiomatic approach is suggested as a way to analyze such theory-selection criteria and judge their reasonability based on finite databases.

[1]  Larry Samuelson,et al.  Subjectivity in Inductive Inference , 2009 .

[2]  A. Kolmogorov.  Three approaches to the quantitative definition of information , 1968 .

[3]  C. S. Wallace,et al.  Statistical and Inductive Inference by Minimum Message Length (Information Science and Statistics) , 2005 .

[4]  Ray J. Solomonoff,et al.  A Formal Theory of Inductive Inference. Part II , 1964, Inf. Control..

[5]  Itzhak Gilboa,et al.  Empirical Similarity , 2004, The Review of Economics and Statistics.

[6]  David L. Dowe,et al.  Minimum Message Length and Kolmogorov Complexity , 1999, Comput. J..

[7]  Dov Samet,et al.  Probabilities as Similarity-Weighted Frequencies , 2004 .

[8]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[9]  H. Akaike.  A new look at the statistical model identification , 1974 .

[10]  A. Shiryayev.  On Tables of Random Numbers , 1993 .

[11]  B. Silverman.  Density estimation for statistics and data analysis , 1986 .

[12]  C. S. Wallace,et al.  An Information Measure for Classification , 1968, Comput. J..

[13]  E. H. Simpson,et al.  The Interpretation of Interaction in Contingency Tables , 1951 .

[14]  L. Wittgenstein.  Tractatus Logico-Philosophicus , 2021, Nordic Wittgenstein Review.

[15]  Gregory J. Chaitin,et al.  On the Length of Programs for Computing Finite Binary Sequences , 1966, JACM.

[16]  Peter Grünwald,et al.  Invited review of the book Statistical and Inductive Inference by Minimum Message Length , 2006 .

[17]  Bruce A. Murtagh,et al.  Computational Experience with Quadratically Convergent Minimisation Methods , 1970, Comput. J..

[18]  Victor Pambuccian,et al.  Simplicity , 1988, Notre Dame J. Formal Log..

[19]  L. J. Savage,et al.  The Foundations of Statistics , 1955 .

[20]  Itzhak Gilboa,et al.  A theory of case-based decisions , 2001 .

[21]  Simon,et al.  A HISTORY OF WESTERN PHILOSOPHY , 2012 .

[22]  N. Goodman.  Fact, Fiction, and Forecast , 1955 .

[23]  I. Gilboa,et al.  Inductive Inference: An Axiomatic Approach , 2001 .

[24]  B. D. Finetti.  La prévision : ses lois logiques, ses sources subjectives , 1937 .