A High-Performance Approach to Model Calibration and Validation

A new model validation approach is presented that integrates parallel processing on high- performance computing clusters with random search algorithms to fit cognitive models to human performance data. The efficiency, accuracy, and non-biasness of this approach surpasses the prevalent manual optimization techniques; results in exceptional model to human data fits; and is available and extendable to other parameterized models, search algorithms, cognitive architectures, and cluster computing resources. Results from testing the validation approach using a prototype cognitive model of a serial subtraction task, the ACT-R cognitive architecture, and 15 individual fits are described.

[1]  Matthias Scheutz,et al.  Combinatorics Meets Processing Power: Large-Scale Computational Resources for BRIMS , 2007 .

[2]  Joseph Lee Rodgers,et al.  Theory development should begin (but not end) with good empirical fits: a comment on Roberts and Pashler (2000). , 2002, Psychological review.

[3]  Frank E. Ritter,et al.  Using A Genetic Algorithm to Optimize the Fit of Cognitive Models , 2004, ICCM.

[4]  Kevin A. Gluck,et al.  Lessons Learned and Future Directions for the AMBR Model Comparison Project , 2001 .

[5]  C. Kirschbaum,et al.  The 'Trier Social Stress Test'--a tool for investigating psychobiological stress responses in a laboratory setting. , 1993, Neuropsychobiology.

[6]  Sue E. Kase,et al.  Modeling the Range of Performance on the Serial Subtraction Task , 2007 .

[7]  H Pashler,et al.  How persuasive is a good fit? A comment on theory testing. , 2000, Psychological review.

[8]  Sue E. Kase.  Parallel genetic algorithm optimization of a cognitive model: investigating group and individual performance on a math stressor task , 2008 .

[9]  A. Newell.  Unified Theories of Cognition , 1990 .

[10]  David E. Goldberg,et al.  Genetic and evolutionary algorithms come of age , 1994, CACM.

[11]  John R. Anderson.  How Can the Human Mind Occur in the Physical Universe , 2007 .

[12]  Joan Ryder,et al.  Validation and calibration of human performance models to support simulation-based acquisition , 2004, Proceedings of the 2004 Winter Simulation Conference, 2004..

[13]  Erick Cantú-Paz,et al.  Efficient and Accurate Parallel Genetic Algorithms , 2000, Genetic Algorithms and Evolutionary Computation.

[14]  Frank E. Ritter,et al.  Lessons from Defining Theories of Stress , 2007 .

[15]  John R. Anderson,et al.  The Adaptive Character of Thought , 1990 .

[16]  Frank E. Ritter,et al.  Towards Fair Comparisons of Connectionist Algorithms through Automatically Optimized Parameter Sets , 1991 .

[17]  J. Ivey,et al.  Ann Arbor, Michigan , 1969 .

[18]  Frank E. Ritter,et al.  The Rise of Cognitive Architectures , 2007, Integrated Models of Cognitive Systems.

[19]  Samuel G. Schiflett,et al.  The Sleepiness-Induced Lapsing and Cognitive Slowing (SILCS) Model: Predicting Fatigue Effects on Warfighter Performance , 2000 .

[20]  D E Kieras,et al.  A computational theory of executive cognitive processes and multiple-task performance: Part 1. Basic mechanisms. , 1997, Psychological review.