Scientists often run experiments to distinguish competing theories. This requires patience, rigor, and ingenuity - there is often a large space of possible experiments one could run. But we need not comb this space by hand - if we represent our theories as formal models and explicitly declare the space of experiments, we can automate the search for good experiments, looking for those with high expected information gain. Here, we present a general and principled approach to experiment design based on probabilistic programming languages (PPLs). PPLs offer a clean separation between declaring problems and solving them, which means that the scientist can automate experiment design by simply declaring her model and experiment spaces in the PPL without having to worry about the details of calculating information gain. We demonstrate our system in two case studies drawn from cognitive psychology, where we use it to design optimal experiments in the domains of sequence prediction and categorization. We find strong empirical validation that our automatically designed experiments were indeed optimal. We conclude by discussing a number of interesting questions for future research.
[1]
W. Wagenaar,et al.
The perception of randomness
,
1991
.
[2]
Andreas Krause,et al.
Near-Optimal Bayesian Active Learning with Noisy Observations
,
2010,
NIPS.
[3]
Peter A. J. Hilbers,et al.
A Bayesian approach to targeted experiment design
,
2012,
Bioinform..
[4]
D. Lindley.
On a Measure of the Information Provided by an Experiment
,
1956
.
[5]
Thomas L. Griffiths,et al.
From Algorithmic to Subjective Randomness
,
2003,
NIPS.
[6]
Michael P. H. Stumpf,et al.
Maximizing the Information Content of Experiments in Systems Biology
,
2013,
PLoS Comput. Biol..
[7]
Jay I. Myung,et al.
Optimal experimental design for model discrimination.
,
2009,
Psychological review.
[8]
Jeannot Trampert,et al.
Optimal nonlinear Bayesian experimental design: an application to amplitude versus offset experiments
,
2003
.
[9]
Xun Huan,et al.
Accelerated Bayesian Experimental Design for Chemical Kinetic Models
,
2010
.
[10]
Douglas L. Medin,et al.
Context theory of classification learning.
,
1978
.