Do Additional Features Help or Hurt Category Learning? The Curse of Dimensionality in Human Learners

The curse of dimensionality, which has been widely studied in statistics and machine learning, occurs when additional features cause the size of the feature space to grow so quickly that learning classification rules becomes increasingly difficult. How do people overcome the curse of dimensionality when acquiring real-world categories that have many different features? Here we investigate the possibility that the structure of categories can help. We show that when categories follow a family resemblance structure, people are unaffected by the presence of additional features in learning. However, when categories are based on a single feature, they fall prey to the curse, and having additional irrelevant features hurts performance. We compare and contrast these results to three different computational models to show that a model with limited computational capacity best captures human performance across almost all of the conditions in both experiments.

[1]  Joshua B. Tenenbaum,et al.  Human-level concept learning through probabilistic program induction , 2015, Science.

[2]  Gregory Piatetsky-Shapiro,et al.  High-Dimensional Data Analysis: The Curses and Blessings of Dimensionality , 2000 .

[3]  John Paul Minda,et al.  The Psychology of Thinking , 2015 .

[4]  R. Nosofsky.  Attention, similarity, and the identification-categorization relationship. , 1986, Journal of experimental psychology. General.

[5]  L. E. Bourne,et al.  Mathematical theory of concept identification. , 1959, Psychological review.

[6]  Adam N Sanborn,et al.  Rational approximations to rational models: alternative algorithms for category learning. , 2010, Psychological review.

[7]  Joshua B. Tenenbaum,et al.  Mapping a Manifold of Perceptual Observations , 1997, NIPS.

[8]  R. Nosofsky.  Exemplar-Based Accounts of Relations Between Classification, Recognition, and Typicality , 1988 .

[9]  Aaron B. Hoffman,et al.  Prior knowledge enhances the category dimensionality effect , 2008, Memory & cognition.

[10]  I. Mclaren,et al.  Perceptual Learning and Free Classification , 1998, The Quarterly journal of experimental psychology. B, Comparative and physiological psychology.

[11]  Eamonn J. Keogh,et al.  Curse of Dimensionality , 2010, Encyclopedia of Machine Learning.

[12]  I. Mclaren,et al.  Perceptual Categorization: Connectionist Modelling and Decision Rules , 1998, The Quarterly journal of experimental psychology. B, Comparative and physiological psychology.

[13]  Patrick Shafto,et al.  Cooperative inference: Features, objects, and collections. , 2016, Psychological review.

[14]  Charles Kemp,et al.  How to Grow a Mind: Statistics, Structure, and Abstraction , 2011, Science.

[15]  J. Tenenbaum,et al.  Structured statistical models of inductive reasoning. , 2009, Psychological review.

[16]  M. Clyde,et al.  Mixtures of g Priors for Bayesian Variable Selection , 2008 .

[17]  R. Roe,et al.  IRRELEVANT INFORMATION IN PROBABILISTIC CATEGORIZATION , 1996 .

[18]  L E BOURNE,et al.  The identification of concepts as a function of amounts of relevant and irrelevant information. , 1961, The American journal of psychology.

[19]  E. Markman.  Categorization and naming in children , 1989 .

[20]  J. D. Smith,et al.  Prototypes in category learning: the effects of category size, category structure, and stimulus complexity. , 2001, Journal of experimental psychology. Learning, memory, and cognition.

[21]  Richard D. Morey,et al.  Baysefactor: Computation of Bayes Factors for Common Designs , 2018 .

[22]  Fraser Milton,et al.  Combination or Differentiation? Two theories of processing order in classification , 2015, Cognitive Psychology.

[23]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[24]  Jeffrey N. Rouder,et al.  Computation of Bayes Factors for Common Designs , 2015 .

[25]  R. Shepard,et al.  Learning and memorization of classifications. , 1961 .

[26]  Nick Chater,et al.  Empiricism and Language Learnability , 2015 .

[27]  Edward E. Smith,et al.  Correlated properties in natural categories , 1984 .

[28]  D. Medin,et al.  SUSTAIN: a network model of category learning. , 2004, Psychological review.

[29]  Thomas L. Griffiths,et al.  A Rational Analysis of Rule-Based Concept Learning , 2008, Cogn. Sci..

[30]  Ron Sun,et al.  The Cambridge Handbook of Computational Psychology , 2008 .

[31]  G. Murphy,et al.  The Big Book of Concepts , 2002 .

[32]  E. M. Wright,et al.  Adaptive Control Processes: A Guided Tour , 1961, The Mathematical Gazette.

[33]  Charles Kemp,et al.  Bayesian models of cognition , 2008 .

[34]  J. Kruschke,et al.  ALCOVE: an exemplar-based connectionist model of category learning. , 1992, Psychological review.

[35]  I. Mclaren,et al.  Generalization in Human Category Learning: A Connectionist Account of Differences in Gradient after Discriminative and Non discriminative Training , 1997 .

[36]  M. Posner,et al.  On the genesis of abstract ideas. , 1968, Journal of experimental psychology.

[37]  Douglas L. Medin,et al.  Context theory of classification learning. , 1978 .

[38]  Geoffrey I. Webb,et al.  Encyclopedia of Machine Learning , 2011, Encyclopedia of Machine Learning.

[39]  Willard Van Orman Quine,et al.  Word and Object , 1960 .

[40]  E. Rosch,et al.  Family resemblances: Studies in the internal structure of categories , 1975, Cognitive Psychology.

[41]  Jonathan D. Cohen,et al.  Learning to selectively attend , 2010 .

[42]  Michel Verleysen,et al.  The Curse of Dimensionality in Data Mining and Time Series Prediction , 2005, IWANN.

[43]  N. Goodman.  Fact, Fiction, and Forecast , 1955 .

[44]  Jana Jarecki,et al.  The Assumption of Class-conditional Independence in Category Learning , 2013, CogSci.

[45]  John R. Anderson,et al.  The Adaptive Character of Thought , 1990 .

[46]  Jonathan D. Nelson,et al.  Naïve and Robust: Class-Conditional Independence in Human Classification Learning , 2018, Cogn. Sci..

[47]  Jeffrey N. Rouder,et al.  Default Bayes factors for ANOVA designs , 2012 .

[48]  Mark Steyvers,et al.  Topics in semantic representation. , 2007, Psychological review.

[49]  Aaron B. Hoffman,et al.  Category Dimensionality and Feature Knowledge: When More Features Are Learned as Easily as Fewer Number of Dimensions in Natural and Experimental Categories , 2022 .

[50]  R. Nosofsky,et al.  Rule-plus-exception model of classification learning. , 1994, Psychological review.