Parts and Wholes: Scenarios and Simulators for Human Performance Studies

As tools like full-scale simulators and microworlds become more readily available to researchers, a fundamental question remains the extent to which full scenarios and simulators are necessary for valid and generalizable results. In this paper, we explore the continuum of scenarios and simulators and evaluate the advantages and disadvantages of each for human performance studies. The types of scenarios presented to participants may range from microtasks to complex multi-step scenarios. Microtasks usually involve only brief exposure to the human-system interface but may thereby facilitate ready data collection through repeated trials. In contrast, full scenarios present a sequence of actions that may require an extended period of time. The tradeoffs center on the fidelity of the situations and the requirements for the type of human performance data to be collected. The type of simulator presented to participants may range from a part-task simulator, to a simplified microworld, or to a full-scope high-fidelity simulator. The simplified simulators present greater opportunity for control but lose much of the context of real-world use found in full-scope simulators. We frame scenarios and simulators in the context of micro- vs. macro-cognition and provide examples of how the different experimental design choices lend themselves to different types of studies.