Blobs, Dipsy-Doodles and Other Funky Things: Framework Anomalies in Exploratory Data Analysis

Blobs, Dipsy-Doodles and Other Funky Things: Framework Anomalies in Exploratory Data Analysis Susan B. Trickett J. Gregory Trafton ( stricket@gmu.edu) ( trafton@itd.nrl.navy.mil ) ( schunn@gmu.edu ) Department of Psychology George Mason University Fairfax, VA 22030 USA Naval Research Laboratory NRL Code 5513 Washington, DC 20375 Department of Psychology George Mason University Fairfax, VA 22030 Abstract This study investigates the role of anomalies in the ex- ploratory analysis of visual scientific data. We found that anomalies played a crucial role as two experts analyzed as- tronomical data. Not only did they pay significantly more attention to anomalies than expected phenomena, both immediately and over time, but also anomalies provided a framework within which they investigated the data. Introduction Attention to the unexpected may be an important component of scientific discovery. Exploring anomalies can lead to the- ory development and even conceptual change. Philosophers of science (e.g., Kuhn, 1962) have argued that unusual find- ings play a key role in scientific revolutions, and scientists themselves have claimed that investigating anomalies lies at the heart of scientific innovation (e.g., Knorr, 1980). Within cognitive psychology, response to anomalous data during scientific inquiry has been noted in a variety of stud- ies, including historical reconstructions of actual scientific discoveries (e.g., Kulkarni & Simon, 1988), on-line studies of scientists (e.g., Dunbar, 1997), laboratory studies in which participants “rediscover” a scientific phenomenon (Dunbar 1993), and studies of those with little scientific training as they perform abstract scientific reasoning tasks (e.g., Tweney, Dowerty, & Mynatt, 1982; Klahr & Dunbar, 1988). These studies have not yielded a consistent pattern of response to unexpected data, possibly because of the range of scientific training and knowledge among the participants. Recognizing this variety of responses to anomalous data, Chinn and Brewer (1992, 1993), propose a taxonomy of seven reactions to unusual findings, from ignoring the data and upholding the theory to accepting the data and changing the theory. This taxonomy is derived from anecdotal exam- ples from the history of science and from empirical studies of scientific reasoning in the psychological literature. Al- though Chinn and Brewer propose that this taxonomy ap- plies to scientists and non-scientists alike, they have tested it only among undergraduates with little scientific training. Thus, despite the general belief that anomalous data is important in scientific discovery, no clear picture has emerged of how scientists (as opposed to laypersons per- forming scaled-down scientific discovery tasks) respond to unexpected findings. On one hand, there is a well-established tradition in studies of scientific thinking that shows people overlook data inconsistent with their hypothesis, looking Christian D. Schunn only for support for their theories (e.g., Wason, 1960). Within this tradition, scientists have been found to be as susceptible to this confirmation bias as laypeople (e.g., Ma- honey & DeMonbreun, 1977; Mitroff, 1974.) Similarly, studies of complex visualization usage have shown that ex- pert meteorologists do not pay much attention to unusual or anomalous features. Instead, they seem to extract informa- tion in a very goal directed manner, rarely following up on features that are not directly relevant to their immediate task (Trafton et al., under review). This evidence—of confirma- tion bias, even among scientists, and of the goal-directed nature of complex visualization usage—suggests that scien- tists may overlook unexpected results or anomalies. On the other hand, however, Dunbar has recently ques- tioned the validity of the studies of confirmation bias on the grounds that they employ arbitrary experimental tasks that involve no scientific knowledge and therefore bear little rela- tionship to tasks that real scientists perform (Dunbar, 1997). Dunbar has argued that in order to investigate how scientists reason, one must observe scientists as they perform their scientific tasks. Using an “in vivo” methodology that involves observing actual scientists at work, Dunbar has suggested that scien- tists do attend to unusual results (Dunbar, 1997). He found not only that scientists attended to unexpected results more than they did to expected findings, but also that individual scientists were quick to discard a hypothesis when faced with results that were inconsistent with it. Furthermore, he noted that in lab meetings, the group of scientists tended to focus on a surprising result until they had constructed a plausible hypothesis to account for it. Dunbar concluded that attending to anomalous findings is an important strategy that contrib- utes to successful scientific inquiry (Dunbar 1997). Simi- larly, Kulkarni and Simon (1988) identified an “attend to surprising result” heuristic as crucial to Hans Krebs' discov- ery of the urea cycle. Both Chinn and Brewer's and Dunbar's studies have in- volved participants, whether trained scientists or not, who were evaluating data to test a specific theory. However, there are many phases of scientific inquiry, and response to anomalous data might be quite different during an explora- tory phase from when a theory is firmly established. During exploratory data analysis, theories may be only partially defined. Nonetheless, given their extensive domain knowl- edge, scientists doubtless have general frameworks which lead to expectations that may or may not be met by the data. They may therefore pay more attention to unusual results,