Measuring Cognitive Abilities of Machines , Humans and Non-Human Animals in a Unified Way : towards Universal Psychometrics

We present and develop the notion of ‘universal psychometrics’ as a subject of study, and eventually a discipline, that focusses on the measurement of cognitive abilities for the machine kingdom, which comprises any kind of individual or collective, either artificial, biological or hybrid. Universal psychometrics can be built, of course, upon the experience, techniques and methodologies from (human) psychometrics, comparative cognition and related areas. Conversely, the perspective and techniques which are being developed in the area of machine intelligence measurement using (algorithmic) information theory can be of much broader applicability and implication outside artificial intelligence. This general approach to universal psychometrics spurs the re-understanding of most (if not all) of the big issues about the measurement of cognitive abilities, and creates a new foundation for (re)defining and mathematically formalising the concept of cognitive task, evaluable subject, interface, task choice, difficulty, agent response curves, etc. We introduce the notion of a universal cognitive test and discuss whether (and when) it may be necessary for exploring the machine kingdom. On the issue of intelligence and very general abilities, we also get some results and connections with the related notions of no-free-lunch theorems and universal priors.

[1]  Peter Grünwald,et al.  Invited review of the book Statistical and Inductive Inference by Minimum Message Length , 2006 .

[2]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[3]  Selmer Bringsjord,et al.  What is Artificial Intelligence? Psychometric AI as an Answer , 2003, IJCAI.

[4]  Jürgen Schmidhuber,et al.  Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990–2010) , 2010, IEEE Transactions on Autonomous Mental Development.

[5]  Stewart W. Wilson.  The animat path to AI , 1991 .

[6]  A. E. Crawley.  Behavior: An Introduction to Comparative Psychology , 1915, Nature.

[7]  M. Bitterman.  Toward a comparative psychology of learning. , 1960 .

[8]  M. Hagberg.  Editorial , 2004 .

[9]  Matthew V. Mahoney,et al.  Text Compression as a Test for Artificial Intelligence , 1999, AAAI/IAAI.

[10]  David L. Dowe,et al.  A Non-Behavioural, Computational Extension to the Turing Test , 1998 .

[11]  José Hernández-Orallo,et al.  On the Computational Measurement of Intelligence Factors , 2011 .

[12]  José Hernández-Orallo,et al.  Beyond the Turing Test , 2000, J. Log. Lang. Inf..

[13]  Ray J. Solomonoff,et al.  A Formal Theory of Inductive Inference. Part I , 1964, Inf. Control..

[14]  Anthony J. Bagnall,et al.  Learning Mazes with Aliasing States: An LCS Algorithm with Associative Perception , 2009, Adapt. Behav..

[15]  S. Shettleworth.  The evolution of comparative cognition: Is the snark still a boojum? , 2009, Behavioural Processes.

[16]  S. Shettleworth.  Cognition, evolution, and behavior , 1998 .

[17]  John Langford,et al.  Telling humans and computers apart automatically , 2004, CACM.

[18]  M. Tomasello,et al.  Humans Have Evolved Specialized Skills of Social Cognition: The Cultural Intelligence Hypothesis , 2007, Science.

[19]  T. G. Evans.  A program for the solution of a class of geometric-analogy intelligence-test questions , 1964 .

[20]  Donald R. Griffin.  Prospects for a Cognitive Ethology , 1978 .

[21]  José Hernández-Orallo,et al.  On Evaluating Agent Performance in a Fixed Period of Time (Extended Version) , 2010, AGI 2010.

[22]  José Hernández-Orallo,et al.  Comparing Humans and AI Agents , 2011, AGI.

[23]  R. Pintner,et al.  Crossroads in the Mind of Man: A Study of Differentiable Mental Abilities. , 1929 .

[24]  M. Tomasello,et al.  The Structure of Individual Differences in the Cognitive Abilities of Children and Chimpanzees , 2010, Psychological science.

[25]  H A SIMON,et al.  HUMAN ACQUISITION OF CONCEPTS FOR SEQUENTIAL PATTERNS. , 1963, Psychological review.

[26]  R. J. Solomon Off,et al.  The time scale of artificial intelligence: Reflections on social effects , 1985 .

[27]  J. Raven,et al.  Manual for Raven's progressive matrices and vocabulary scales , 1962 .

[28]  E. Thorndike.  The law of effect. , 1927 .

[29]  J. Strassmann,et al.  Absence of within–colony kin discrimination in behavioural interactions of swarm–founding wasps , 1997, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[30]  Douglas K. Detterman.  A challenge to Watson , 2011 .

[31]  H. Simon,et al.  What makes some problems really hard: Explorations in the problem space of difficulty , 1990, Cognitive Psychology.

[32]  Shane Legg,et al.  Universal Intelligence: A Definition of Machine Intelligence , 2007, Minds and Machines.

[33]  Cyril Burt,et al.  The structure of human abilities , 1951 .

[34]  Robert J. Sternberg.  Handbook of Intelligence: Handbook of Intelligence , 2000 .

[35]  C. Q. Lee,et al.  The Computer Journal , 1958, Nature.

[36]  J. Hernández-Orallo,et al.  IQ tests are not for machines, yet , 2012 .

[37]  S. Shettleworth.  Where Is the Comparison in Comparative Cognition? Alternative Research Programs , 1993 .

[38]  B. Beck,et al.  Chimpocentrism: Bias in cognitive ethology , 1982 .

[39]  D. Deutsch.  Quantum theory, the Church–Turing principle and the universal quantum computer , 1985, Proceedings of the Royal Society of London. A. Mathematical and Physical Sciences.

[40]  Frank S. Freeman,et al.  Interpretation of educational measurements , 1928 .

[41]  D. Borsboom.  Measuring the mind: Conceptual issues in contemporary psychometrics , 2005 .

[42]  R. Solomonoff.  Does Algorithmic Probability Solve the Problem of Induction , 2006 .

[43]  Alejandra Barrera,et al.  Comparative Experimental Studies on Spatial Memory and Learning in Rats and Robots , 2011, J. Intell. Robotic Syst..

[44]  David L. Dowe,et al.  MML clustering of multi-state, Poisson, von Mises circular and Gaussian distributions , 2000, Stat. Comput..

[45]  E. Boring.  Intelligence as the Tests Test It. , 1961 .

[46]  David L. Dowe,et al.  A computational extension to the Turing test , 1997 .

[47]  David L. Dowe,et al.  Minimum message length and generalized Bayesian nets with asymmetric languages , 2005 .

[48]  David H. Wolpert,et al.  Coevolutionary free lunches , 2005, IEEE Transactions on Evolutionary Computation.

[49]  Ben Goertzel,et al.  AGI Preschool: A Framework for Evaluating Early-Stage Human-like AGIs , 2009 .

[50]  Elizabeth M. Brannon,et al.  How does cognition evolve? Phylogenetic comparative psychology , 2012, Animal Cognition.

[51]  José Hernández-Orallo,et al.  Thesis: Computational measures of information gain and reinforcement in inference processes , 2000, AI Commun..

[52]  Chris S. Wallace.  Physically random generator , 1990, Comput. Syst. Sci. Eng..

[53]  C. S. Wallace,et al.  An Information Measure for Classification , 1968, Comput. J..

[54]  S. Embretson,et al.  Item response theory for psychologists , 2000 .

[55]  David J. Weiss,et al.  Better Data From Better Measurements Using Computerized Adaptive Testing , 2011 .

[56]  David L. Dowe,et al.  The Turing Test , 2003 .

[57]  R. Cattell,et al.  Age differences in fluid and crystallized intelligence. , 1967, Acta psychologica.

[58]  H. Harlow,et al.  Learning motivated by a manipulation drive. , 1950, Journal of experimental psychology.

[59]  Jean-Arcady Meyer,et al.  Adaptive Behavior , 2005 .

[60]  D. Borsboom,et al.  The concept of validity. , 2004, Psychological review.

[61]  D. Eignor.  The standards for educational and psychological testing. , 2013 .

[62]  D. Dowe,et al.  A conceptual model for integrating physical geography research and coastal wetland management, with an Australian example , 2010 .

[63]  A. M. Turing,et al.  Computing Machinery and Intelligence , 1950, The Philosophy of Artificial Intelligence.

[64]  David L. Dowe,et al.  Intrinsic classification by MML - the Snob program , 1994 .

[65]  J. Guilford,et al.  The nature of human intelligence. , 1968 .

[66]  C. Adcock,et al.  Primary Mental Abilities. , 1971, The Journal of general psychology.

[67]  I. J. Good,et al.  Speculations Concerning the First Ultraintelligent Machine , 1965, Adv. Comput..

[68]  José Hernández-Orallo,et al.  Compression and Intelligence: Social Environments and Communication , 2011, AGI.

[69]  José Hernández-Orallo,et al.  Measuring universal intelligence: Towards an anytime intelligence test , 2010, Artif. Intell..

[70]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[71]  David L. Dowe,et al.  MML, hybrid Bayesian network graphical models, statistical consistency, invarianc , 2010 .

[72]  Bill Hibbard.  Bias and No Free Lunch in Formal Measures of Intelligence , 2009, J. Artif. Gen. Intell..

[73]  Ming Li,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 1997, Texts in Computer Science.

[74]  C. Burt,et al.  The structure of the mind; a review of the results of factor analysis. , 1949, The British journal of educational psychology.

[75]  Ming Li,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 2019, Texts in Computer Science.

[76]  Leonid A. Levin,et al.  Average Case Complete Problems , 1986, SIAM J. Comput..

[77]  David L. Dowe,et al.  A computer program capable of passing I.Q. tests , 2008 .

[78]  Michael Tomasello,et al.  Primate Cognition , 2010, Top. Cogn. Sci..

[79]  Michael C. Pyryt.  Human cognitive abilities: A survey of factor analytic studies , 1998 .

[80]  Georg Rasch,et al.  Probabilistic Models for Some Intelligence and Attainment Tests , 1981, The SAGE Encyclopedia of Research Design.

[81]  José Hernández-Orallo,et al.  On More Realistic Environment Distributions for Defining, Evaluating and Developing Intelligence , 2011, AGI.

[82]  C. Spearman.  General intelligence Objectively Determined and Measured , 1904 .

[83]  David L. Dowe,et al.  Minimum Message Length and Kolmogorov Complexity , 1999, Comput. J..

[84]  David L. Dowe,et al.  Bayes not Bust! Why Simplicity is no Problem for Bayesians1 , 2007, The British Journal for the Philosophy of Science.

[85]  H. J. Eysenck,et al.  The structure and measurement of intelligence , 2004, Naturwissenschaften.

[86]  David L. Dowe,et al.  General Bayesian networks and asymmetric languages , 2003 .

[87]  A. Jensen,et al.  The g factor , 1996, Nature.

[88]  B. Jack Copeland,et al.  The Church-Turing Thesis , 2007 .

[89]  J. Cantlon,et al.  Shared System for Ordering Small and Large Numbers in Monkeys and Humans , 2006, Psychological science.

[90]  G. Chaitin.  Gödel's theorem and information , 1982 .