Universal psychometrics: Measuring cognitive abilities in the machine kingdom

We present and develop the notion of 'universal psychometrics' as a subject of study, and eventually a discipline, that focusses on the measurement of cognitive abilities for the machine kingdom, which comprises any (cognitive) system, individual or collective, either artificial, biological or hybrid. Universal psychometrics can be built, of course, upon the experience, techniques and methodologies from (human) psychometrics, comparative cognition and related areas. Conversely, the perspective and techniques which are being developed in the area of machine intelligence measurement using (algorithmic) information theory can be of much broader applicability and implication outside artificial intelligence. This general approach to universal psychometrics spurs the re-understanding of most (if not all) of the big issues about the measurement of cognitive abilities, and creates a new foundation for (re)defining and mathematically formalising the concept of cognitive task, evaluable subject, interface, task choice, difficulty, agent response curves, etc. We introduce the notion of a universal cognitive test and discuss whether (and when) it may be necessary for exploring the machine kingdom. On the issue of intelligence and very general abilities, we also get some results and connections with the related notions of no-free-lunch theorems and universal priors.

[1]  Ben Goertzel,et al.  AGI Preschool: A Framework for Evaluating Early-Stage Human-like AGIs , 2009 .

[2]  P. Kleingeld,et al.  The Stanford Encyclopedia of Philosophy , 2013 .

[3]  M. Tomasello,et al.  The Structure of Individual Differences in the Cognitive Abilities of Children and Chimpanzees , 2010, Psychological science.

[4]  A. M. Turing,et al.  Computing Machinery and Intelligence , 1950, The Philosophy of Artificial Intelligence.

[5]  Ray J. Solomonoff,et al.  A Formal Theory of Inductive Inference. Part I , 1964, Inf. Control..

[6]  D. Borsboom,et al.  The concept of validity. , 2004, Psychological review.

[7]  Ray J. Solomonofi.  INDUCTIVE INFERENCE RESEARCH STATUS, SPRING 1967 , 1967 .

[8]  A. Newell.  You can't play 20 questions with nature and win : projective comments on the papers of this symposium , 1973 .

[9]  E. Boring.  Intelligence as the Tests Test It. , 1961 .

[10]  C. S. Wallace,et al.  An Information Measure for Classification , 1968, Comput. J..

[11]  R. Pintner,et al.  Crossroads in the Mind of Man: A Study of Differentiable Mental Abilities. , 1929 .

[12]  José Hernández-Orallo,et al.  Beyond the Turing Test , 2000, J. Log. Lang. Inf..

[13]  D. Borsboom.  Measuring the mind: Conceptual issues in contemporary psychometrics , 2005 .

[14]  D. Deutsch.  Quantum theory, the Church–Turing principle and the universal quantum computer , 1985, Proceedings of the Royal Society of London. A. Mathematical and Physical Sciences.

[15]  Ray J. Solomonoff,et al.  Complexity-based induction systems: Comparisons and convergence theorems , 1978, IEEE Trans. Inf. Theory.

[16]  B. Jack Copeland,et al.  The Church-Turing Thesis , 2007 .

[17]  David L. Dowe,et al.  A computer program capable of passing I.Q. tests , 2008 .

[18]  B. Beck,et al.  Chimpocentrism: Bias in cognitive ethology , 1982 .

[19]  John B. Watson,et al.  Behavior : An Introduction to Comparative Psychology , 2006 .

[20]  David L. Dowe,et al.  Minimum Message Length and Kolmogorov Complexity , 1999, Comput. J..

[21]  Jürgen Schmidhuber,et al.  Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990–2010) , 2010, IEEE Transactions on Autonomous Mental Development.

[22]  S. Gosling.  From mice to men: what can we learn about personality from animal research? , 2001, Psychological bulletin.

[23]  C. S. Wallace,et al.  Statistical and Inductive Inference by Minimum Message Length (Information Science and Statistics) , 2005 .

[24]  Ray J. Solomonoff,et al.  A Formal Theory of Inductive Inference. Part II , 1964, Inf. Control..

[25]  Matthew V. Mahoney,et al.  Text Compression as a Test for Artificial Intelligence , 1999, AAAI/IAAI.

[26]  M. Bitterman.  Toward a comparative psychology of learning. , 1960 .

[27]  S. Shettleworth.  Where Is the Comparison in Comparative Cognition? Alternative Research Programs , 1993 .

[28]  David L. Dowe,et al.  The Turing Test , 2003 .

[29]  Ming Li,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 1997, Texts in Computer Science.

[30]  M. Fox,et al.  The 3rd International Planning Competition: Results and Analysis , 2003, J. Artif. Intell. Res..

[31]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[32]  José Hernández-Orallo,et al.  On Potential Cognitive Abilities in the Machine Kingdom , 2013, Minds and Machines.

[33]  C. Spearman.  General intelligence Objectively Determined and Measured , 1904 .

[34]  Donald E. Knuth,et al.  Sorting and Searching , 1973 .

[35]  D. Cliff,et al.  Complexity and emergent behaviour in ICT systems , 2004 .

[36]  G. Chaitin.  Gödel's theorem and information , 1982 .

[37]  José Hernández-Orallo,et al.  On Evaluating Agent Performance in a Fixed Period of Time (Extended Version) , 2010, AGI 2010.

[38]  Andrew G. Barto,et al.  Reinforcement learning , 1998 .

[39]  R. J. Solomonoff,et al.  The time scale of artificial intelligence: Reflections on social effects , 1985 .

[40]  Alejandra Barrera,et al.  Comparative Experimental Studies on Spatial Memory and Learning in Rats and Robots , 2011, J. Intell. Robotic Syst..

[41]  H. Eysenck.  PRIMARY MENTAL ABILITIES , 1939 .

[42]  A. Jensen,et al.  The g factor , 1996, Nature.

[43]  Shane Legg,et al.  Universal Intelligence: A Definition of Machine Intelligence , 2007, Minds and Machines.

[44]  J. Strassmann,et al.  Absence of within–colony kin discrimination in behavioural interactions of swarm–founding wasps , 1997, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[45]  Elizabeth M. Brannon,et al.  How does cognition evolve? Phylogenetic comparative psychology , 2012, Animal Cognition.

[46]  Georg Rasch,et al.  Probabilistic Models for Some Intelligence and Attainment Tests , 1981, The SAGE Encyclopedia of Research Design.

[47]  José Hernández-Orallo,et al.  Measuring universal intelligence: Towards an anytime intelligence test , 2010, Artif. Intell..

[48]  Alija Kulenović,et al.  Standards for Educational and Psychological Testing , 1999 .

[49]  B. Skinner,et al.  The Behavior of Organisms: An Experimental Analysis , 2016 .

[50]  H A SIMON,et al.  HUMAN ACQUISITION OF CONCEPTS FOR SEQUENTIAL PATTERNS. , 1963, Psychological review.

[51]  Selmer Bringsjord,et al.  What is Artificial Intelligence? Psychometric AI as an Answer , 2003, IJCAI.

[52]  M. Tomasello,et al.  Humans Have Evolved Specialized Skills of Social Cognition: The Cultural Intelligence Hypothesis , 2007, Science.

[53]  José Hernández-Orallo,et al.  On More Realistic Environment Distributions for Defining, Evaluating and Developing Intelligence , 2011, AGI.

[54]  John Langford,et al.  Telling humans and computers apart automatically , 2004, CACM.

[55]  Douglas K. Detterman.  A challenge to Watson , 2011 .

[56]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[57]  J. Carroll.  Human Cognitive Abilities-a sur-vey of factor-analytic studies , 1993 .

[58]  J. Guilford,et al.  The nature of human intelligence. , 1968 .

[59]  H. Simon,et al.  What makes some problems really hard: Explorations in the problem space of difficulty , 1990, Cognitive Psychology.

[60]  I. J. Good,et al.  Speculations Concerning the First Ultraintelligent Machine , 1965, Adv. Comput..

[61]  David J. Weiss,et al.  Better Data From Better Measurements Using Computerized Adaptive Testing , 2011 .

[62]  L. Thurstone.  Primary mental abilities. , 1938, Science.

[63]  Jean-Arcady Meyer,et al.  From Animals to Animats: Proceedings of the International Conference on Simulation of Adaptive Behavior (1st) Held in Paris, France on 24-28 September 1990 , 1991 .

[64]  David L. Dowe,et al.  A computational extension to the Turing test , 1997 .

[65]  Anthony J. Bagnall,et al.  Learning Mazes with Aliasing States: An LCS Algorithm with Associative Perception , 2009, Adapt. Behav..

[66]  Richard P. Brent,et al.  Some Comments on C. S. Wallace's Random Number Generators , 2008, Comput. J..

[67]  David L. Dowe,et al.  Foreword re C. S. Wallace , 2008, Comput. J..

[68]  Barbara Webb,et al.  Animals Versus Animats: Or Why Not Model the Real Iguana? , 2009, Adapt. Behav..

[69]  Chris S. Wallace.  Physically random generator , 1990, Comput. Syst. Sci. Eng..

[70]  J. Cantlon,et al.  Shared System for Ordering Small and Large Numbers in Monkeys and Humans , 2006, Psychological science.

[71]  H. J. Eysenck,et al.  The structure and measurement of intelligence , 2004, Naturwissenschaften.

[72]  C. Burt,et al.  The structure of the mind; a review of the results of factor analysis. , 1949, The British journal of educational psychology.

[73]  Gregory R. Parr,et al.  Comparative experimental studies of dental implants , 1990 .

[74]  Shimon Whiteson,et al.  The Reinforcement Learning Competitions , 2010 .

[75]  Cyril Burt,et al.  The structure of human abilities , 1951 .

[76]  Peter Grünwald,et al.  Invited review of the book Statistical and Inductive Inference by Minimum Message Length , 2006 .

[77]  E. Thorndike.  The law of effect. , 1927 .

[78]  José Hernández-Orallo,et al.  Thesis: Computational measures of information gain and reinforcement in inference processes , 2000, AI Commun..

[79]  Howard Wainer,et al.  Computerized Adaptive Testing: A Primer , 2000 .

[80]  David L. Dowe,et al.  A Non-Behavioural, Computational Extension to the Turing Test , 1998 .

[81]  Michael R. Genesereth,et al.  General Game Playing: Overview of the AAAI Competition , 2005, AI Mag..

[82]  R. Sternberg,et al.  Handbook of Intelligence , 2000 .

[83]  H. Harlow,et al.  Learning motivated by a manipulation drive. , 1950, Journal of experimental psychology.

[84]  Bill Hibbard.  Bias and No Free Lunch in Formal Measures of Intelligence , 2009, J. Artif. Gen. Intell..

[85]  David L. Dowe,et al.  MML, hybrid Bayesian network graphical models, statistical consistency, invarianc , 2010 .

[86]  José Hernández-Orallo,et al.  On the Computational Measurement of Intelligence Factors , 2011 .

[87]  J. Hernández-Orallo,et al.  IQ tests are not for machines, yet , 2012 .

[88]  S. Shettleworth.  Cognition, evolution, and behavior , 1998 .

[89]  José Hernández-Orallo,et al.  Compression and Intelligence: Social Environments and Communication , 2011, AGI.

[90]  Selmer Bringsjord,et al.  Psychometric artificial intelligence , 2011, J. Exp. Theor. Artif. Intell..

[91]  J. Raven,et al.  Manual for Raven's progressive matrices and vocabulary scales , 1962 .

[92]  Stewart W. Wilson.  The animat path to AI , 1991 .

[93]  Apostolos Syropoulos,et al.  Hypercomputation: Computing Beyond the Church-Turing Barrier , 2008 .

[94]  R. Cattell,et al.  Age differences in fluid and crystallized intelligence. , 1967, Acta psychologica.

[95]  Leonid A. Levin,et al.  Average Case Complete Problems , 1986, SIAM J. Comput..

[96]  Shane Legg,et al.  An Approximation of the Universal Intelligence Measure , 2011, Algorithmic Probability and Friends.

[97]  José Hernández-Orallo,et al.  Comparing Humans and AI Agents , 2011, AGI.

[98]  Frank S. Freeman,et al.  Interpretation of educational measurements , 1928 .

[99]  Ming Li,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 2019, Texts in Computer Science.

[100]  R. Solomonoff.  Does Algorithmic Probability Solve the Problem of Induction , 2006 .

[101]  S. Shettleworth.  The evolution of comparative cognition: Is the snark still a boojum? , 2009, Behavioural Processes.

[102]  Donald R. Griffin.  Prospects for a Cognitive Ethology , 1978 .

[103]  Michael Thielscher,et al.  General Game Playing , 2015 .

[104]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[105]  David H. Wolpert,et al.  Coevolutionary free lunches , 2005, IEEE Transactions on Evolutionary Computation.

[106]  S. Embretson,et al.  Item response theory for psychologists , 2000 .

[107]  Donald Ervin Knuth,et al.  The Art of Computer Programming , 1968 .