On the universality of cognitive tests

The analysis of the adaptive behaviour of many different kinds of systems such as humans, animals and machines, requires more general ways of assessing their cognitive abilities. This need is strengthened by increasingly more tasks being analysed for and completed by a wider diversity of systems, including swarms and hybrids. The notion of universal test has recently emerged in the context of machine intelligence evaluation as a way to define and use the same cognitive test for a variety of systems, using some principled tasks and adapting the interface to each particular subject. However, how far can universal tests be taken? This paper analyses this question in terms of subjects, environments, space-time resolution, rewards and interfaces. This leads to a number of findings, insights and caveats, according to several levels where universal tests may be progressively more difficult to conceive, implement and administer. One of the most significant contributions is given by the realisation that more universal tests are defined as maximisations of less universal tests for a variety of configurations. This means that universal tests must be necessarily adaptive.

[1]  E Donchin,et al.  A metric for thought: a comparison of P300 latency and reaction time. , 1981, Science.

[2]  José Hernández-Orallo,et al.  Evaluating a Reinforcement Learning Algorithm with a General Intelligence Test , 2011, CAEPIA.

[3]  T. Matsuzawa,et al.  Working memory of numerals in chimpanzees , 2007, Current Biology.

[4]  José Hernández-Orallo,et al.  Turing Tests with Turing Machines , 2012, Turing-100.

[5]  Lotfi A. Zadeh,et al.  A fuzzy-algorithmic approach to the definition of complex or imprecise concepts , 1976 .

[6]  McCay Vernon,et al.  Psychological Evaluation and Testing of Children Who Are Deaf-Blind. , 1979 .

[7]  Z. Zenn Bien,et al.  Machine intelligence quotient: its measurements and applications , 2002, Fuzzy Sets Syst..

[8]  David L. Dowe,et al.  A Non-Behavioural, Computational Extension to the Turing Test , 1998 .

[9]  José Hernández-Orallo,et al.  Turing machines and recursive Turing Tests , 2012 .

[10]  S. Levinson,et al.  The myth of language universals: language diversity and its importance for cognitive science. , 2009, The Behavioral and brain sciences.

[11]  Michael R. Genesereth,et al.  General Game Playing: Overview of the AAAI Competition , 2005, AI Mag..

[12]  R. Sternberg,et al.  Handbook of Intelligence , 2000 .

[13]  José Hernández-Orallo,et al.  Universal psychometrics: Measuring cognitive abilities in the machine kingdom , 2014, Cognitive Systems Research.

[14]  Michael Thielscher,et al.  General Game Playing , 2015 .

[15]  Ray J. Solomonoff,et al.  A Formal Theory of Inductive Inference. Part I , 1964, Inf. Control..

[16]  Ray J. Solomonofi.  INDUCTIVE INFERENCE RESEARCH STATUS, SPRING 1967 , 1967 .

[17]  José Hernández-Orallo.  A (hopefully) Unbiased Universal Environment Class for Measuring Intelligence of Biological and Artificial Systems , 2009, AGI 2010.

[18]  James Rogers,et al.  Animal Pattern-Learning Experiments: Some Mathematical Background∗ , 2006 .

[19]  Danesh Tarapore,et al.  Quantifying patterns of agent-environment interaction , 2006, Robotics Auton. Syst..

[20]  Shane T. Mueller,et al.  Adapting the Turing Test for Embodied Neurocognitive Evaluation of Biologically-Inspired Cognitive Agents , 2008, AAAI Fall Symposium: Biologically Inspired Cognitive Architectures.

[21]  Randall D. Beer,et al.  Information Dynamics of Evolved Agents , 2010, SAB.

[22]  José Hernández-Orallo,et al.  Compression and Intelligence: Social Environments and Communication , 2011, AGI.

[23]  A R MAHRER.  Potential intelligence: a learning theory approach to description and clinical implication. , 1958, The Journal of general psychology.

[24]  David L. Dowe,et al.  A computer program capable of passing I.Q. tests , 2008 .

[25]  Mathew Iredale.  Who are you calling “bird brain”? , 2006 .

[26]  Selmer Bringsjord,et al.  Psychometric artificial intelligence , 2011, J. Exp. Theor. Artif. Intell..

[27]  David L. Dowe,et al.  Minimum Message Length and Kolmogorov Complexity , 1999, Comput. J..

[28]  José Hernández-Orallo.  A short note on estimating intelligence from user profiles in the context of universal psychometrics: prospects and caveats , 2013, ArXiv.

[29]  Douglas K. Detterman.  A challenge to Watson , 2011 .

[30]  H A SIMON,et al.  HUMAN ACQUISITION OF CONCEPTS FOR SEQUENTIAL PATTERNS. , 1963, Psychological review.

[31]  Clara Mancini,et al.  Animal-computer interaction: a manifesto , 2011, INTR.

[32]  Mark Buchanan.  Learning from bacteria , 2008 .

[33]  A. M. Turing,et al.  Computing Machinery and Intelligence , 1950, The Philosophy of Artificial Intelligence.

[34]  Peter Grünwald,et al.  Invited review of the book Statistical and Inductive Inference by Minimum Message Length , 2006 .

[35]  Matt Jones,et al.  The BICA Cognitive Decathlon: A Test Suite for Biologically-Inspired Cognitive Agents , 2007 .

[36]  Philip Hingston,et al.  A new design for a Turing Test for Bots , 2010, Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games.

[37]  Barbara C. Scholz,et al.  For universals (but not finite-state learning) visit the zoo , 2009, Behavioral and Brain Sciences.

[38]  José Hernández-Orallo,et al.  Thesis: Computational measures of information gain and reinforcement in inference processes , 2000, AI Commun..

[39]  Hiroaki Kitano,et al.  RoboCup: The Robot World Cup Initiative , 1997, AGENTS '97.

[40]  Ray J. Solomonoff,et al.  Algorithmic Probability, Heuristic Programming and AGI , 2010, AGI 2010.

[41]  M. Fox,et al.  The 3rd International Planning Competition: Results and Analysis , 2003, J. Artif. Intell. Res..

[42]  B. Jack Copeland,et al.  The Turing Test* , 2000, Minds and Machines.

[43]  Francisco Calvo Garzón.  Nonclassical connectionism should enter the decathlon , 2003, Behavioral and Brain Sciences.

[44]  B. Bassler,et al.  Interspecies communication in bacteria. , 2003, The Journal of clinical investigation.

[45]  José Hernández-Orallo.  Constructive reinforcement learning , 2000, Int. J. Intell. Syst..

[46]  José Hernández-Orallo,et al.  On Measuring Social Intelligence: Experiments on Competition and Cooperation , 2012, AGI.

[47]  J. Hernández-Orallo,et al.  IQ tests are not for machines, yet , 2012 .

[48]  David L. Dowe,et al.  A computational extension to the Turing test , 1997 .

[49]  Shane T. Mueller,et al.  Is the Turing Test Still Relevant ? A Plan for Developing the Cognitive Decathlon to Test Intelligent Embodied Behavior , 2008 .

[50]  S. Shettleworth.  Cognition, evolution, and behavior , 1998 .

[51]  S. Shettleworth.  Fundamentals of Comparative Cognition , 2012 .

[52]  Paul Schweizer,et al.  The Truly Total Turing Test* , 1998, Minds and Machines.

[53]  Bill Hibbard.  Bias and No Free Lunch in Formal Measures of Intelligence , 2009, J. Artif. Gen. Intell..

[54]  Pat Langley,et al.  Artificial Intelligence and Cognitive Systems , 2011 .

[55]  T. G. Evans.  A program for the solution of a class of geometric-analogy intelligence-test questions , 1964 .

[56]  José Hernández-Orallo,et al.  On the Computational Measurement of Intelligence Factors , 2011 .

[57]  Matthew V. Mahoney,et al.  Text Compression as a Test for Artificial Intelligence , 1999, AAAI/IAAI.

[58]  Timothy J. O'Donnell,et al.  Evolutionary Linguistics: A New Look at an Old Landscape , 2007 .

[59]  Erik Borg,et al.  A review and evaluation of research on the deaf-blind from perceptual, communicative, social and rehabilitative perspectives , 2001, Scandinavian audiology.

[60]  Shimon Whiteson,et al.  The Reinforcement Learning Competitions , 2010 .

[61]  T R THORP,et al.  Predicting potential intelligence. , 1959, Journal of clinical psychology.

[62]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[63]  Shane Legg,et al.  Algorithmic probability , 2007, Scholarpedia.

[64]  Thomas G. Evans,et al.  A heuristic program to solve geometric-analogy problems , 1964, AFIPS '64 (Spring).

[65]  Fred Keijzer,et al.  Plants: Adaptive behavior, root-brains, and minimal cognition , 2011, Adapt. Behav..

[66]  Jennifer Chu-Carroll,et al.  Building Watson: An Overview of the DeepQA Project , 2010, AI Mag..

[67]  Murray Campbell,et al.  Deep Blue , 2002, Artif. Intell..

[68]  José Hernández-Orallo,et al.  On More Realistic Environment Distributions for Defining, Evaluating and Developing Intelligence , 2011, AGI.

[69]  John Langford,et al.  Telling humans and computers apart automatically , 2004, CACM.

[70]  D. Borsboom.  Measuring the mind: Conceptual issues in contemporary psychometrics , 2005 .

[71]  José Hernández-Orallo,et al.  Measuring Cognitive Abilities of Machines , Humans and Non-Human Animals in a Unified Way : towards Universal Psychometrics , 2012 .

[72]  P. Madsen,et al.  Dolphin whistles: a functional misnomer revealed by heliox breathing , 2012, Biology Letters.

[73]  José Hernández-Orallo,et al.  Complexity distribution of agent policies , 2013, ArXiv.

[74]  Kent G. Bailey,et al.  Potential Intelligence or Intelligence Test Potential?: A Question of Empirical Validity. , 1972 .

[75]  C. Q. Lee,et al.  The Computer Journal , 1958, Nature.

[76]  Ray J. Solomonoff,et al.  A Formal Theory of Inductive Inference. Part II , 1964, Inf. Control..

[77]  J. Hernández-Orallo,et al.  Potential Properties of Turing Machines , 2012 .

[78]  K. Zuberbühler,et al.  Interspecies semantic communication in two forest primates , 2000, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[79]  Paul Arnold,et al.  Tactile memory of deaf-blind adults on four tasks. , 2002, Scandinavian journal of psychology.

[80]  H. Mar,et al.  Psychological Evaluation of Children Who Are Deaf-Blind: An Overview with Recommendations for Practice. , 1996 .

[81]  José Hernández-Orallo,et al.  On Potential Cognitive Abilities in the Machine Kingdom , 2013, Minds and Machines.

[82]  G. Chaitin.  Gödel's theorem and information , 1982 .

[83]  Martin Ziegler,et al.  Variations of the Turing Test in the Age of Internet and Virtual Reality , 2009, KI.

[84]  Timothy Q. Gentner,et al.  Recursive syntactic pattern learning by songbirds , 2006, Nature.

[85]  Marvin Minsky,et al.  Semantic Information Processing , 1968 .

[86]  Randall D. Beer,et al.  Autopoiesis and Cognition in the Game of Life , 2004, Artificial Life.

[87]  C. Lebiere,et al.  The Newell Test for a theory of cognition , 2003, Behavioral and Brain Sciences.

[88]  P R Sanberg,et al.  "Neural capacity" in Mimosa pudica: a review. , 1976, Behavioral biology.

[89]  Viktor Zárský,et al.  Plant intelligence , 2009, Plant signaling & behavior.

[90]  Shane Legg,et al.  An Approximation of the Universal Intelligence Measure , 2011, Algorithmic Probability and Friends.

[91]  Shane Legg,et al.  Universal Intelligence: A Definition of Machine Intelligence , 2007, Minds and Machines.

[92]  José Hernández-Orallo,et al.  Measuring universal intelligence: Towards an anytime intelligence test , 2010, Artif. Intell..

[93]  E. Boring.  Intelligence as the Tests Test It. , 1961 .

[94]  C. S. Wallace,et al.  An Information Measure for Classification , 1968, Comput. J..

[95]  José Hernández-Orallo,et al.  Comparing Humans and AI Agents , 2011, AGI.

[96]  Celeste Biever.  Ultimate IQ: one test to rule them all , 2011 .

[97]  Selmer Bringsjord,et al.  What is Artificial Intelligence? Psychometric AI as an Answer , 2003, IJCAI.

[98]  Roger T Hanlon,et al.  Intense ultrasonic clicks from echolocating toothed whales do not elicit anti–predator responses or debilitate the squid Loligo pealeii , 2007, Biology Letters.

[99]  John M Beggs,et al.  Partial information decomposition as a spatiotemporal filter. , 2011, Chaos.

[100]  Itamar Arel,et al.  Beyond the Turing Test , 2009, Computer.

[101]  R. Solomonoff.  INDUCTIVE INFERENCE RESEARCH STATUS , SPRING 1967 , 1966 .

[102]  Trevor Bekolay,et al.  A Large-Scale Model of the Functioning Brain , 2012, Science.

[103]  Charles Twardy,et al.  Refining the cognitive decathlon , 2008, PerMIS.

[104]  H. Simon,et al.  What makes some problems really hard: Explorations in the problem space of difficulty , 1990, Cognitive Psychology.