Situated Breakdown Analysis for the Evaluation of a Virtual Environment

It is a widely-supported tenet in human-computer interaction that the meaningful unit of analysis is not the technical device alone, but the technical device together with the person interacting with it; the reason is that what is a relevant property of a technology is only understandable with respect to the specific goals and resources activated during its usage. This basic reflection should also inspire the procedure followed to evaluate the usability of a technology, namely its efficiency and satisfaction for a specific class of users. The topic of this paper is precisely to describe a method developed in compliance with this observation and aimed at evaluating the usability of virtual environments. Two main requirements were set forth: first, the method should take the strong connection between humans and technology as its building block, by linking a property of the virtual environment to a particular use that makes that property relevant. To this goal, action has been placed at the center of the analysis; the functional properties of the VE are then observed in the general economy of users’ interaction with the technology and the whole ensemble is the appropriate object of evaluation. Such ‘action-based’ approach (Gamberini, Spagnolli, 2002) is reminiscent of the Situated Action theory (Suchman, 1987) and Activity Theory (Nardi, 1996); the former proposes a detailed analysis of the sequential interaction with the technology and provides a rich examination of the structure given to it by the users. The latter focuses more on specific phenomena, such as contradictions and breakdowns, identified by the evaluators; it allows to profit from data poor in comments and verbalizations, and to analyze the interaction with the technology from a structural and organizational level. As a second requisite for the method, we wanted it to benefit from the advantages of both approaches; thus we decided to concentrate on the breakdowns occurring during users’ interaction with the VE but to study these episodes from a situated point of view. In our definition, breakdowns reveal an inappropriate interpretation of the possibilities for action offered by the virtual environment and are to be analyzed in their sequential, contextual unfolding. This version of breakdown analysis highlights the spontaneous, subjective problems in the use of a technology and connects them to specific aspects of users’ action. It renews the ergonomic tradition of error studies (Reason, 1990; Rasmussen, 1980) with an ethnographic contamination, that pays attention to users’ contextualized practices. It also suits the kind of data the interaction with a virtual environment is mostly made of, namely bodily action in a three-dimensional space. Few methods with these characteristics have been employed so far to analyze the interaction with the VE. After a brief introduction, the paper will describe the basics of this approach and illustrate them with instances from the evaluation of a virtual library.