The User's Mental Model of an Information Retrieval System: An Experiment on a Prototype Online Catalog

Abstract An empirical study was performed to train naive subjects in the use of a prototype Boolean logic-based information retrieval system on a database of bibliographic records. The research was based on the mental models theory which proposes that people can be trained to develop a “mental model” or a qualitative simulation of a system which will aid in generating methods for interacting with the system, debugging errors, and keeping track of one's place in the system. It follows that conceptual training based on a system model will be superior to procedural training based on the mechanics of the system. We performed a laboratory experiment with two training conditions (model and procedural), and with each condition split by sex. Forty-three subjects participated in the experiment, but only 32 were able to reach the minimum competency level required to complete the experiment. The data analysis incorporated time-stamped monitoring data, personal characteristics variables, affective variables, and interview data in which subjects described how they thought the system worked (an articulation of the model). As predicted, the model-based training had no effect on the ability to perform simple, procedural tasks, but subjects trained with a model performed better on complex tasks that required extrapolation from the basic operations of the system. A stochastic process analysis of search-state transitions reinforced this conclusion. Subjects had difficulty articulating a model of the system, and we found no differences in articulation by condition. The high number of subjects (26% ) who were unable to pass the benchmark test indicates that the retrieval tasks were inherently difficult. More interestingly, those who dropped out were significantly more likely to be humanities or social science majors than science or engineering majors, suggesting important individual differences and equity issues. The sex-related differences were slight, although significant, and suggest future research questions.