Practical Evaluation of Integrated Cognitive Systems

This paper argues that integration is an essential approach to advancing the state of the art in cognitive systems. Integration is the path to aggregating requirements for cognition, as well as the solutions to those requirements. However, evaluation of integrated cognitive systems has proven difficult and expensive, which has hindered scientific advances. We suggest approaches to evaluating integrated cognitive systems by introducing the notion of practical evaluation, as well as the imposition of requirements that can only be satisfied with cognitive systems solutions. Practical evaluation involves not only measuring a system’s task competence but also the properties of adaptivity, directability, understandability, and trustworthiness.

[1]  John E. Laird,et al.  The Soar Cognitive Architecture , 2012 .

[2]  A. Damasio.  Self comes to mind : constructing the conscious brain , 2010 .

[3]  Kurt VanLehn,et al.  A model of the self-explanation effect. , 1992 .

[4]  Selmer Bringsjord,et al.  What is Artificial Intelligence? Psychometric AI as an Answer , 2003, IJCAI.

[5]  Ben Goertzel,et al.  Proceedings of the Second Conference on Artificial General Intelligence , 2009 .

[6]  Randolph M. Jones,et al.  Constraints on the Design of a High-Level Model of Cognition , 1999 .

[7]  Randolph M. Jones,et al.  Automated Intelligent Pilots for Combat Flight Simulation , 1998, AI Mag..

[8]  Pat Langley,et al.  Machine Learning for Intelligent Systems , 1997, AAAI/IAAI.

[9]  C. Lebiere,et al.  The Atomic Components of Thought , 1998 .

[10]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[11]  C. Lebiere,et al.  Metrics for Cognitive Architecture Evaluation , 2007 .

[12]  N. Chater,et al.  Proceedings of the fourteenth annual conference of the cognitive science society , 1992 .

[13]  Richard Reviewer-Granger.  Unified Theories of Cognition , 1991, Journal of Cognitive Neuroscience.

[14]  Pat Langley,et al.  Cognitive architectures: Research issues and challenges , 2009, Cognitive Systems Research.

[15]  Pat Langley,et al.  Claims and Challenges in Evaluating Human-Level Intelligent Systems , 2009 .

[16]  Beverly Park Woolf,et al.  Building Intelligent Interactive Tutors: Student-centered Strategies for Revolutionizing E-learning , 2008 .

[17]  Michael H. Bowling,et al.  Towards robust teams with many agents , 2002, AAMAS '02.

[18]  Scott A. Wallace,et al.  Behavior Bounding: Toward Effective Comparisons of Agents & Humans , 2003, IJCAI.

[19]  C. Lebiere,et al.  The Newell Test for a theory of cognition , 2003, Behavioral and Brain Sciences.