Designing Good Experiments to Test Bad Hypotheses

Abstract : What does it take to design a good experiment? Given an hypothesis to be evaluated -- either in isolation or in competition with alternatives -- what formal rules, heuristics, and pragmatic constraints combine to yield a potentially informative experiment? How do subjects' expectations about the plausibility of an hypothesis effect the kind of experiments that they design, their ability to accurately observe and encode experimental outcomes and their responses to information that is consistent or inconsistent with the hypothesis? In this paper, we address these questions by creating a simulated discovery context and examining how subjects go about designing experiments to test hypotheses that are always, at the outset, incorrect. Thirty-six adult subjects were trained on the basic functions of a programmable robot. Then they were presented with a new function key ( a 'repeat' key) and asked to find out how it worked. Our results show that subjects are remarkably adept at designing and interpreting experiments in a novel domain. When subjects are given a plausible hypothesis, they tend to design an experiment that demonstrates the effect that is to be expected. When given inplausible hypothesis, they write programs that are good discriminators.