The Generalized Universal Law of Generalization

Abstract It has been argued by Shepard that there is a robust psychological law that relates the distance between a pair of items in psychological space and the probability that they will be perceived as similar. Specifically, this probability is a negative exponential function of the distance between the pair of items. In experimental contexts, distance is typically defined in terms of a multidimensional space—but this assumption seems unlikely to hold for complex stimuli. We show that, nonetheless, the Universal Law of Generalization can be derived in the more complex setting of arbitrary stimuli, using a much more universal measure of distance. This universal distance is defined as the length of the shortest program that transforms the representations of the two items of interest into one another: The algorithmic information distance. It is universal in the sense that it minorizes every computable distance: It is the smallest computable distance. We show that the Universal Law of Generalization holds with probability going to one—provided the probabilities concerned are computable. We also give a mathematically more appealing form of the Universal Law.

[1]  D M Ennis,et al.  Toward a universal law of generalization. , 1988, Science.

[2]  J. Fodor,et al.  Connectionism and cognitive architecture: A critical analysis , 1988, Cognition.

[3]  Nick Chater,et al.  Simplicity and the mind , 1997 .

[4]  F. Attneave,et al.  The determination of perceived tridimensional orientation by minimum criteria , 1969 .

[5]  Frank Restle,et al.  Theory of Serial Pattern Learning: Structural Trees. , 1970 .

[6]  E. Leeuwenberg.  Quantitative specification of information in sequential patterns. , 1969, Psychological review.

[7]  Rudolf Carnap,et al.  The continuum of inductive methods , 1952 .

[8]  G. Ekman.  Dimensions of Color Vision , 1954 .

[9]  Herbert A. Simon,et al.  Complexity and the representation of patterned sequences of symbols. , 1972 .

[10]  David M. Raup,et al.  How Nature Works: The Science of Self-Organized Criticality , 1997 .

[11]  R. Shepard,et al.  Monotone mapping of similarities into a general metric space , 1974 .

[12]  I. J. Myung,et al.  Toward a method of selecting among computational models of cognition. , 2002, Psychological review.

[13]  Peter A. van der Helm,et al.  Simplicity versus likelihood in visual perception: from surprisals to precisals. , 2000 .

[14]  Milton Friedman,et al.  Essays in Positive Economics , 1954 .

[15]  Per Martin-Löf,et al.  The Definition of Random Sequences , 1966, Inf. Control..

[16]  I. Biederman.  Recognition-by-components: a theory of human image understanding. , 1987, Psychological review.

[17]  J. Wagemans,et al.  Detection of visual symmetries. , 1995, Spatial vision.

[18]  Roger C. Schank,et al.  SCRIPTS, PLANS, GOALS, AND UNDERSTANDING , 1988 .

[19]  N. Chater,et al.  Rational models of cognition , 1998 .

[20]  John G. Kemeny,et al.  The Use of Simplicity in Induction , 1953 .

[21]  H. Simon,et al.  Theories of Decision-Making in Economics and Behavioural Science , 1966 .

[22]  G. A. Miller,et al.  An Analysis of Perceptual Confusions Among Some English Consonants , 1955 .

[23]  R. Shepard,et al.  Learning and memorization of classifications. , 1961 .

[24]  F. Restle.  Coding theory of the perception of motion configurations. , 1979, Psychological review.

[25]  Paul M. B. Vitányi,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 1993, Graduate Texts in Computer Science.

[26]  G. Seth.  Psychology of Language , 1968, Nature.

[27]  John A. Goldsmith,et al.  Unsupervised Learning of the Morphology of a Natural Language , 2001, CL.

[28]  Ming Li,et al.  Minimum description length induction, Bayesianism, and Kolmogorov complexity , 1999, IEEE Trans. Inf. Theory.

[29]  N. Chater,et al.  Rational models of cognition , 1998 .

[30]  Refractor.  Vision , 2000, The Lancet.

[31]  N. Guttman,et al.  Discriminability and stimulus generalization. , 1956, Journal of experimental psychology.

[32]  Nick Chater,et al.  Representational Distortion, Similarity and the Universal Law of Generalization , 1997 .

[33]  J. Tenenbaum,et al.  Generalization, similarity, and Bayesian inference. , 2001, The Behavioral and brain sciences.

[34]  J. Hochberg,et al.  A quantitative approach to figural "goodness". , 1953, Journal of experimental psychology.

[35]  R. Nosofsky.  Overall similarity and the identification of separable-dimension stimuli: A choice model analysis , 1985, Perception & psychophysics.

[36]  H A SIMON,et al.  HUMAN ACQUISITION OF CONCEPTS FOR SEQUENTIAL PATTERNS. , 1963, Psychological review.

[37]  F. Bartlett,et al.  Remembering: A Study in Experimental and Social Psychology , 1932 .

[38]  E. Leeuwenberg,et al.  Goodness of visual regularities: a nontransformational approach. , 1996, Psychological review.

[39]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[40]  Paul C. Vitz,et al.  A coded element model of the perceptual processing of sequential stimuli. , 1969 .

[41]  Roger N. Shepard,et al.  Maximum Entropy Inference and Stimulus Generalization , 1996 .

[42]  R. Nosofsky.  Exemplar-Based Accounts of Relations Between Classification, Recognition, and Typicality , 1988 .

[43]  P. Odifreddi.  Classical recursion theory , 1989 .

[44]  E. Rothkopf.  A measure of stimulus similarity and errors in some paired-associate learning tasks. , 1957, Journal of experimental psychology.

[45]  J. Rissanen.  Stochastic Complexity and Modeling , 1986 .

[46]  Douglas L. Medin,et al.  Context theory of classification learning. , 1978 .

[47]  F Boselie,et al.  Against the likelihood principle in visual form perception. , 1988, Psychological review.

[48]  Ernst Mach,et al.  The Analysis of Sensations. , 1916 .

[49]  K. Koffka.  Principles Of Gestalt Psychology , 1936 .

[50]  Ming Li,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 2019, Texts in Computer Science.

[51]  Janet L. Yellen,et al.  Can Small Deviations from Rationality Make Significant Differences to Economic Equilibria , 1985 .

[52]  Zenon W. Pylyshyn,et al.  Connectionism and cognitive architecture , 1993 .

[53]  S S Stevens,et al.  To Honor Fechner and Repeal His Law: A power function, not a log function, describes the operating characteristic of a sensory system. , 1961, Science.

[54]  A. Turing.  On Computable Numbers, with an Application to the Entscheidungsproblem. , 1937 .

[55]  Ray J. Solomonoff,et al.  Complexity-based induction systems: Comparisons and convergence theorems , 1978, IEEE Trans. Inf. Theory.

[56]  A. Tversky,et al.  Spatial versus tree representations of proximity data , 1982 .

[57]  T. A. Cartwright,et al.  Distributional regularity and phonotactic constraints are useful for segmentation , 1996, Cognition.

[58]  R. Nosofsky.  Similarity, frequency, and category representations. , 1988 .

[59]  Clifford Konold,et al.  Making Sense of Randomness " Implicit Encoding as a Basis for Judgment , 1997 .

[60]  I. J. Myung,et al.  Counting probability distributions: Differential geometry and model selection , 2000, Proc. Natl. Acad. Sci. USA.

[61]  Jorma Rissanen,et al.  Fisher information and stochastic complexity , 1996, IEEE Trans. Inf. Theory.

[62]  R. Shepard,et al.  Toward a universal law of generalization for psychological science. , 1987, Science.

[63]  E. Leeuwenberg,et al.  Coding theory of visual pattern completion. , 1981, Journal of experimental psychology. Human perception and performance.

[64]  N. Chater,et al.  Similarity as transformation , 2003, Cognition.

[65]  E. Leeuwenberg.  A perceptual coding language for visual and auditory patterns. , 1971, The American journal of psychology.

[66]  Xin Chen,et al.  An information-based sequence distance and its application to whole mitochondrial genome phylogeny , 2001, Bioinform..

[67]  Jacob Feldman,et al.  Minimization of Boolean complexity in human concept learning , 2000, Nature.

[68]  J. Fodor,et al.  The Psychology of Language , 1974 .

[69]  Nick Chater,et al.  Similarity: A Transformational Approach , 2001 .

[70]  Nick Chater,et al.  A simplicity principle in unsupervised human categorization , 2002, Cogn. Sci..

[71]  John R. Anderson.  The Adaptive Character of Thought , 1990 .

[72]  C. S. Wallace,et al.  An Information Measure for Classification , 1968, Comput. J..

[73]  Ming Li,et al.  Applying MDL to learn best model granularity , 2000, Artif. Intell..

[74]  Péter Gács,et al.  Algorithmic statistics , 2000, IEEE Trans. Inf. Theory.

[75]  Gustav Theodor Fechner,et al.  Elements of psychophysics , 1966 .

[76]  Robert L. Goldstone,et al.  Relational similarity and the nonindependence of features in similarity judgments , 1991, Cognitive Psychology.

[77]  Nick Chater,et al.  Reconciling simplicity and likelihood principles in perceptual organization. , 1996, Psychological review.

[78]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[79]  K Cheng,et al.  Shepard's Universal Law Supported by Honeybees in Spatial Generalization , 2000, Psychological science.

[80]  Ming Li,et al.  Applying MDL to Learning Best Model Granularity , 2000, ArXiv.

[81]  I. J. Myung,et al.  Applying Occam’s razor in modeling cognition: A Bayesian approach , 1997 .

[82]  A. Tversky.  Features of Similarity , 1977 .

[83]  E. Rolls.  High-level vision: Object recognition and visual cognition, Shimon Ullman. MIT Press, Bradford (1996), ISBN 0 262 21013 4 , 1997 .

[84]  N. Chater.  The Search for Simplicity: A Fundamental Cognitive Principle? , 1999 .

[85]  W. Mcguire.  A multiprocess model for paired-associate learning. , 1961 .

[86]  Erhard Tornier,et al.  Grundlagen der Wahrscheinlichkeitsrechnung , 1933 .

[87]  Nick Chater,et al.  The effect of category variability in perceptual categorization. , 2002, Journal of experimental psychology. Learning, memory, and cognition.

[88]  E. Polak,et al.  System Theory , 1963 .

[89]  N. Perrin,et al.  Varieties of perceptual independence. , 1986, Psychological review.

[90]  Conner Sl.  The search for simplicity. , 1994 .

[91]  Ray J. Solomonoff,et al.  A Formal Theory of Inductive Inference. Part II , 1964, Inf. Control..

[92]  C. S. Wallace,et al.  Estimation and Inference by Compact Coding , 1987 .

[93]  H. Kyburg,et al.  How the laws of physics lie , 1984 .