Adaptive Design Optimization in Experiments with People

In cognitive science, empirical data collected from participants are the arbiters in model selection. Model discrimination thus depends on designing maximally informative experiments. It has been shown that adaptive design optimization (ADO) allows one to discriminate models as efficiently as possible in simulation experiments. In this paper we use ADO in a series of experiments with people to discriminate the Power, Exponential, and Hyperbolic models of memory retention, which has been a long-standing problem in cognitive science, providing an ideal setting in which to test the application of ADO for addressing questions about human cognition. Using an optimality criterion based on mutual information, ADO is able to find designs that are maximally likely to increase our certainty about the true model upon observation of the experiment outcomes. Results demonstrate the usefulness of ADO and also reveal some challenges in its implementation.

[1]  L. Verde,et al.  On model selection forecasting, Dark Energy and modified gravity , 2007 .

[2]  J. Wixted,et al.  On the Form of Forgetting , 1991 .

[3]  D. Rubin,et al.  One Hundred Years of Forgetting : A Quantitative Description of Retention , 1996 .

[4]  Peter Grünwald,et al.  A tutorial introduction to the minimum description length principle , 2004, ArXiv.

[5]  Jay I. Myung,et al.  Optimal experimental design for model discrimination. , 2009, Psychological review.

[6]  Robert J. Butera,et al.  Sequential Optimal Design of Neurophysiology Experiments , 2009, Neural Computation.

[7]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[8]  John Brown.  Some Tests of the Decay Theory of Immediate Memory , 1958 .

[9]  Seth D. Guikema,et al.  Formulating informative, data-based priors for failure probability estimation in reliability analysis , 2007, Reliab. Eng. Syst. Saf..

[10]  P. Müller,et al.  Optimal Bayesian Design by Inhomogeneous Markov Chain Simulation , 2004 .

[11]  Bradley P. Carlin,et al.  BAYES AND EMPIRICAL BAYES METHODS FOR DATA ANALYSIS , 1996, Stat. Comput..

[12]  Tuomas J. Lukka,et al.  Bayesian adaptive estimation: The next dimension , 2006 .

[13]  Mark A. Pitt,et al.  Adaptive Design Optimization: A Mutual Information-Based Approach to Model Discrimination in Cognitive Science , 2010, Neural Computation.

[14]  L. R. Peterson,et al.  Short-term retention of individual verbal items. , 1959, Journal of experimental psychology.

[15]  Jay I. Myung,et al.  Assessing the distinguishability of models and the informativeness of data , 2004, Cognitive Psychology.

[16]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[17]  Christian P. Robert,et al.  Bayesian-Optimal Design via Interacting Particle Systems , 2006 .

[18]  Michael D. Lee,et al.  A Bayesian analysis of retention functions , 2004 .

[19]  A. Rukhin.  Bayes and Empirical Bayes Methods for Data Analysis , 1997 .

[20]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .

[21]  J. Wixted,et al.  Genuine power curves in forgetting: A quantitative analysis of individual subject forgetting functions , 1997, Memory & cognition.

[22]  K. Chaloner,et al.  Bayesian Experimental Design: A Review , 1995 .

[23]  Mark A. Pitt,et al.  Better data with fewer participants and trials: improving experiment efficiency with adaptive design optimization , 2009 .

[24]  Peter Green,et al.  Markov chain Monte Carlo in Practice , 1996 .