Affective evaluation of a mobile multimodal dialogue system using brain signals

We propose the use of affective metrics such as excitement, frustration and engagement for the evaluation of multimodal dialogue systems. The affective metrics are elicited from the ElectroEncephaloGraphy (EEG) signals using the Emotiv EPOC neuroheadset device. The affective metrics are used in conjunction with traditional evaluation metrics (turn duration, input modality) to investigate the effect of speech recognition errors and modality usage patterns in a multimodal (touch and speech) dialogue form-filling application for the iPhone mobile device. Results show that: (1) engagement is higher for touch input, while excitement and frustration is higher for speech input, and (2) speech recognition errors and associated repairs correspond to specific dynamic patters of excitement and frustration. Use of such physiological channels and their elaborated interpretation is a challenging but also a potentially rewarding direction towards emotional and cognitive assessment of multimodal interaction design.

[1]  T. Sejnowski,et al.  Removing electroencephalographic artifacts by blind source separation. , 2000, Psychophysiology.

[2]  R. Davidson.  What does the prefrontal cortex “do” in affect: perspectives on frontal EEG asymmetry research , 2004, Biological Psychology.

[3]  Alexandros Potamianos,et al.  Multimodal system evaluation using modality efficiency and synergy metrics , 2008, ICMI '08.

[4]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[5]  Michael E. Smith,et al.  Neurophysiological measures of cognitive workload during human-computer interaction , 2003 .

[6]  Fang Chen,et al.  Galvanic skin response (GSR) as an index of cognitive load , 2007, CHI Extended Abstracts.

[7]  P. Wilson,et al.  The Nature of Emotions , 2012 .

[8]  Klaus-Robert Müller,et al.  The Berlin Brain-Computer Interface , 2008, WCCI.

[9]  R Chavarriaga,et al.  Learning From EEG Error-Related Potentials in Noninvasive Brain-Computer Interfaces , 2010, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[10]  F. Paas,et al.  Cognitive Load Measurement as a Means to Advance Cognitive Load Theory , 2003 .

[11]  Shaohan Hu,et al.  NeuroPhone: brain-mobile phone interface using a wireless EEG headset , 2010, MobiHeld '10.

[12]  Alexandros Potamianos,et al.  A Study in Efficiency and Modality Usage in Multimodal Form Filling Systems , 2008, IEEE Transactions on Audio, Speech, and Language Processing.

[13]  Desney S. Tan,et al.  Feasibility and pragmatics of classifying working memory load with an electroencephalograph , 2008, CHI.

[14]  P. Lang.  The emotion probe. Studies of motivation and attention. , 1995, The American psychologist.

[15]  Roddy Cowie,et al.  FEELTRACE: an instrument for recording perceived emotion in real time , 2000 .