A Tutorial on Adaptive Design Optimization.

Experimentation is ubiquitous in the field of psychology and fundamental to the advancement of its science, and one of the biggest challenges for researchers is designing experiments that can conclusively discriminate the theoretical hypotheses or models under investigation. The recognition of this challenge has led to the development of sophisticated statistical methods that aid in the design of experiments and that are within the reach of everyday experimental scientists. This tutorial paper introduces the reader to an implementable experimentation methodology, dubbed Adaptive Design Optimization, that can help scientists to conduct "smart" experiments that are maximally informative and highly efficient, which in turn should accelerate scientific discovery in psychology and beyond.

[1]  David J. Weiss,et al.  APPLICATION OF COMPUTERIZED ADAPTIVE TESTING TO EDUCATIONAL PROBLEMS , 1984 .

[2]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[3]  K. Chaloner,et al.  Bayesian Experimental Design: A Review , 1995 .

[4]  Christian P. Robert,et al.  Monte Carlo Statistical Methods , 2005, Springer Texts in Statistics.

[5]  R. Hambleton,et al.  Fundamentals of Item Response Theory , 1991 .

[6]  Jonathan D. Nelson,et al.  Experience Matters , 2010, Psychological science.

[7]  Peter Grünwald,et al.  A tutorial introduction to the minimum description length principle , 2004, ArXiv.

[8]  D. Berry.  Bayesian clinical trials , 2006, Nature Reviews Drug Discovery.

[9]  Jay I. Myung,et al.  Optimal experimental design for model discrimination. , 2009, Psychological review.

[10]  Warren B. Powell,et al.  Approximate Dynamic Programming - Solving the Curses of Dimensionality , 2007 .

[11]  Ulrich W. Thonemann,et al.  Optimizing simulated annealing schedules with genetic programming , 1996 .

[12]  J. Timmer,et al.  Systems biology: experimental design , 2009, The FEBS journal.

[13]  Cornelis A.W. Glas,et al.  Computerized adaptive testing : theory and practice , 2000 .

[14]  Nando de Freitas,et al.  An Introduction to MCMC for Machine Learning , 2004, Machine Learning.

[15]  bak gwansu,et al.  An Adaptive Approach to , 2006 .

[16]  T. Loredo.  Bayesian Adaptive Exploration , 2004, astro-ph/0409386.

[17]  L. Reder,et al.  A mechanistic account of the mirror effect for word frequency: a computational model of remember-know judgments in a continuous recognition paradigm. , 2000, Journal of experimental psychology. Learning, memory, and cognition.

[18]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[19]  Jay I. Myung,et al.  Model discrimination through adaptive experimentation , 2011, Psychonomic bulletin & review.

[20]  Daniel R. Cavagnaro,et al.  Discriminating among probability weighting functions using adaptive design optimization , 2013, Journal of risk and uncertainty.

[21]  Mark A. Pitt,et al.  Advances in Minimum Description Length: Theory and Applications , 2005 .

[22]  L. Wasserman,et al.  Computing Bayes Factors by Combining Simulation and Asymptotic Approximations , 1997 .

[23]  J. Wixted,et al.  On the Form of Forgetting , 1991 .

[24]  Sean M. Polyn,et al.  A context maintenance and retrieval model of organizational processes in free recall. , 2009, Psychological review.

[25]  D. Rubin,et al.  One Hundred Years of Forgetting : A Quantitative Description of Retention , 1996 .

[26]  P. Müller,et al.  Optimal Bayesian Design by Inhomogeneous Markov Chain Simulation , 2004 .

[27]  Luis A. Lesmes,et al.  Bayesian adaptive estimation of threshold versus contrast external noise functions: The quick TvC method , 2006, Vision Research.

[28]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[29]  T. Louis,et al.  Bayes and Empirical Bayes Methods for Data Analysis. , 1997 .

[30]  Mark A. Pitt,et al.  Optimal Decision Stimuli for Risky Choice Experiments: An Adaptive Approach , 2013, Manag. Sci..

[31]  S. Lewandowsky,et al.  Forgetting in immediate serial recall: decay, temporal distinctiveness, or interference? , 2008, Psychological review.

[32]  David R. Anderson,et al.  Model selection and multimodel inference : a practical information-theoretic approach , 2003 .

[33]  Bradley P. Carlin,et al.  BAYES AND EMPIRICAL BAYES METHODS FOR DATA ANALYSIS , 1996, Stat. Comput..

[34]  Peter Grünwald,et al.  Accumulative prediction error and the selection of time series models , 2006 .

[35]  Zhong-Lin Lu,et al.  Bayesian adaptive estimation of the contrast sensitivity function: the quick CSF method. , 2010, Journal of vision.

[36]  Jonathan D. Nelson.  Finding useful questions: on Bayesian diagnosticity, probability, impact, and information gain. , 2005, Psychological review.

[37]  C. Tyler,et al.  Bayesian adaptive estimation of psychometric slope and threshold , 1999, Vision Research.

[38]  L. Wasserman,et al.  The Selection of Prior Distributions by Formal Rules , 1996 .

[39]  Tuomas J. Lukka,et al.  Bayesian adaptive estimation: The next dimension , 2006 .

[40]  D. D. Bickerstaff,et al.  Computerized Adaptive Testing , 1989 .

[41]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .

[42]  William J. Hill,et al.  Discrimination Among Mechanistic Models , 1967 .

[43]  Timothy J. Robinson,et al.  Sequential Monte Carlo Methods in Practice , 2003 .

[44]  Anthony C. Atkinson,et al.  Optimum Experimental Designs , 1992 .

[45]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[46]  Petros Dellaportas,et al.  An Introduction to MCMC , 2003 .

[47]  Christian P. Robert,et al.  Bayesian-Optimal Design via Interacting Particle Systems , 2006 .

[48]  A. Rukhin.  Bayes and Empirical Bayes Methods for Data Analysis , 1997 .

[49]  Mark M. Tanaka,et al.  Sequential Monte Carlo without likelihoods , 2007, Proceedings of the National Academy of Sciences.

[50]  Christian P. Robert,et al.  The Bayesian choice : from decision-theoretic foundations to computational implementation , 2007 .

[51]  David A. Cohn,et al.  Improving generalization with active learning , 1994, Machine Learning.

[52]  David A. Cohn,et al.  Active Learning with Statistical Models , 1996, NIPS.

[53]  L. M. M.-T..  Theory of Probability , 1929, Nature.

[54]  Brandon M. Turner,et al.  A tutorial on approximate Bayesian computation , 2012 .

[55]  E. Vul,et al.  Functional adaptive sequential testing. , 2010, Seeing and perceiving.

[56]  A. Atkinson,et al.  Optimal design : Experiments for discriminating between several models , 1975 .

[57]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[58]  A. Pettitt,et al.  Introduction to MCMC , 2012 .

[59]  Mark A. Pitt,et al.  Adaptive Design Optimization in Experiments with People , 2009, NIPS.

[60]  J. Kruschke.  Doing Bayesian Data Analysis: A Tutorial with R and BUGS , 2010 .

[61]  Alan B. Cobo-Lewis,et al.  An adaptive psychophysical method for subject classification , 1997, Perception & psychophysics.

[62]  I. J. Myung,et al.  Toward a method of selecting among computational models of cognition. , 2002, Psychological review.

[63]  Mark A. Pitt,et al.  Adaptive Design Optimization: A Mutual Information-Based Approach to Model Discrimination in Cognitive Science , 2010, Neural Computation.

[64]  D. Lindley.  On a Measure of the Information Provided by an Experiment , 1956 .

[65]  Peter F Thall,et al.  Bayesian adaptive model selection for optimizing group sequential clinical trials , 2008, Statistics in medicine.

[66]  Robert J. Butera,et al.  Sequential Optimal Design of Neurophysiology Experiments , 2009, Neural Computation.

[67]  P. Laycock,et al.  Optimum Experimental Designs , 1995 .

[68]  P. Grünwald,et al.  Catching up faster by switching sooner: a predictive approach to adaptive estimation with an application to the AIC–BIC dilemma , 2012 .

[69]  Edward A. Vessel,et al.  Beauty and the beholder: highly individual taste for abstract, but not real-world images. , 2010, Journal of vision.

[70]  Jay I. Myung,et al.  Optimal Inference and Feedback for Representational Change , 2010 .

[71]  P. Müller.  Simulation Based Optimal Design , 2005 .