Evaluating affective interactions: Alternatives to asking what users feel

In this paper, we advocate the use of behavior-based methods for use in evaluating affective interactions. We consider behavior-based measures to include both measures of bodily movements or physiological signals and task-based performance measures. INTRODUCTION Recent years have seen a large increase in research directed towards adding an affective component to human computer interaction. The ability to measure user affect has become important for intelligent interfaces that aim to either establish believable interactions or alter internal behavior based on the user’s affect. Evaluating and interpreting this measure presents a challenge because of many ambiguities related to affect definition, communication, and interpretation. Classical methods for evaluating affect tend to focus on questionnaires: asking you what you feel now, or interviews, perhaps after the experiment, with a video of your performance in front of you, asking you instant by instant to recall what you felt at each moment during the earlier task. While such “self report” methods are valuable, and we continue to use them in our work, this paper will highlight some alternatives to self-report of feelings. The discussion below is divided into two categories: body measures (e.g. changes in muscle activity), and task measures (e.g. better ability to solve a creative problem). BODY MEASURES OF AFFECT The last decade has brought great strides in giving computers affective perceptual abilities, with new sensors for physiology and for behavior, such as body-worn accelerometers, rubber and fabric electrodes, miniature cameras and microphones, and garment or accessory-type devices, along with new algorithms for recognizing patterns in the sensed signals such as recognition of facial expressions from video or of stress patterns from thermal imagery of the face and other physiological measures. Body measures are not presented here as a replacement for other measures, but rather as additional information that may help combat some of the difficulties encountered with questionnaires and other more subjective methods. Possibly the biggest advantage is that body measurements can be taken in parallel with the interaction rather than interrupting the user or asking him after the task. An exhaustive list of body-based measures is beyond the scope of this paper, however, Table 1 cites a sample of existing methods (leaving out lots of examples of publications in each of these categories, and also leaving out categories, e.g. EEG and ECG-based measures, and more). Clearly there are lots of possible body measures that may capture aspects of an affective state, including the combination of multiple modalities, which can reduce the uncertainty associated with using a single measure (Mednick et al. 1964; DeSilva et al. 1997; Huang et al. 1998; Picard et al. 2001; Kapoor et al. 2004) One benefit of these “body” measures is that they can provide additional insight into the user’s emotional state without directly relying on his cognitive judgment of his emotional state. Additionally, some of them can be used without the user’s knowledge, perhaps with the goal of limiting the amount of misinformation that may arise from his feeling of being monitored. (This can also be seen as a drawback if one is concerned about privacy and about the use of sensing without a person’s knowledge). LEAVE BLANK THE LAST 2.5 cm (1”) OF THE LEFT COLUMN ON THE FIRST PAGE FOR THE COPYRIGHT NOTICE. Table 1. Body-based measures of affect (partial set of examples) Modality Sensor Is it socially communicated? Comments Facial Activity Video (Tian et al. 200; Barlett et al. 1999; Donato et al. 1999; Cowie et al. 2001) Yes Facial expressions can differ significantly from genuinely felt feelings IR Video (Kapoor et al. 2003) Highlights pupils & specularities Usually works better than ordinary video when head moves (better eye detection) Thermal Video (Pavlidis et al. 2002) No Being explored to detect stress and other changes related to deception and frustration Posture Activity Force sensitive resistors (Smith 2000; Mota P Tan et al. 2003) Yes, but not as pressure Good results discriminating level of interest in students in computer learning interactions Hand Tension & Activity Force sensitive resistors or Sentograph (Clynes 1986; Reynolds 2001; Qi P Dennerlein et al. 2003) Varies; depends on gesture Can be sensed from handling of mouse, steering wheel, etc., and pressure has been shown to be higher during a frustrating task