A digital signal processing approach for affective sensing of a computer user through pupil diameter monitoring

OF THE DISSERTATION A DIGITAL SIGNAL PROCESSING APPROACH FOR AFFECTIVE SENSING OF A COMPUTER USER THROUGH PUPIL DIAMETER MONITORING by Ying Gao Florida International University, 2009 Miami, Florida Professor Armando Barreto, Major Professor Recent research has indicated that the pupil diameter (PD) in humans varies with their affective states. However, this signal has not been fully investigated for affective sensing purposes in human-computer interaction systems. This may be due to the dominant separate effect of the pupillary light reflex (PLR), which shrinks the pupil when light intensity increases. In this dissertation, an adaptive interference canceller (AIC) system using the H∞ time-varying (HITV) adaptive algorithm was developed to minimize the impact of the PLR on the measured pupil diameter signal. The modified pupil diameter (MPD) signal, obtained from the AIC was expected to reflect primarily the pupillary affective responses (PAR) of the subject. Additional manipulations of the AIC output resulted in a processed MPD (PMPD) signal, from which a classification feature, PMPDmean, was extracted. This feature was used to train and test a support vector machine (SVM), for the identification of stress states in the subject from whom the pupil diameter signal was recorded, achieving an accuracy rate of 77.78%.

[1]  S. Folkman,et al.  Stress, appraisal, and coping , 1974 .

[2]  John T. Cacioppo,et al.  Psychophysiological approaches to the evaluation of psychotherapeutic process and outcome, 1991 : contributions from social psychophysiology , 1991 .

[3]  William Stafford Noble,et al.  Support vector machine classification on the web , 2004, Bioinform..

[4]  Valery A. Petrushin,et al.  Emotion recognition in speech signal: experimental study, development, and application , 2000, INTERSPEECH.

[5]  Richard S. Lazarus,et al.  The Relationship Between Autonomic Indicators of Psychological Stress: Heart Rate and Skin Conductance , 1963 .

[6]  Paul D. Gamlin,et al.  Characteristics of the pupillary light reflex in the alert rhesus monkey. , 2003, Journal of neurophysiology.

[7]  J. May,et al.  Eye movement indices of mental workload. , 1990, Acta psychologica.

[8]  E. Granholm,et al.  Pupillary responses index cognitive resource limitations. , 1996, Psychophysiology.

[9]  Frank Dellaert,et al.  Recognizing emotion in speech , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[10]  A. Bentivoglio,et al.  Analysis of blink rate patterns in normal subjects , 1997, Movement disorders : official journal of the Movement Disorder Society.

[11]  Robert D. Ward,et al.  Affective computing: problems, reactions and intentions , 2004, Interact. Comput..

[12]  J. Ridley.  Studies of Interference in Serial Verbal Reactions , 2001 .

[13]  Naphtali Rishe,et al.  Significance of Pupil Diameter Measurements for the Assessment of Affective State in Computer Users , 2007 .

[14]  Eva Hudlicka,et al.  To feel or not to feel: The role of affect in human-computer interaction , 2003, Int. J. Hum. Comput. Stud..

[15]  C. MacLeod,et al.  Discrimination of threat cues without awareness in anxiety states. , 1986, Journal of abnormal psychology.

[16]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[17]  Paul C. Bressloff,et al.  Spontaneous oscillations in a nonlinear delayed-feedback shunting model of the pupil light reflex , 1998 .

[18]  Thorsten Joachims,et al.  Text Categorization with Support Vector Machines: Learning with Many Relevant Features , 1998, ECML.

[19]  K. Hugdahl.  Psychophysiology: The Mind-Body Perspective , 1996 .

[20]  Armando Barreto,et al.  Non-intrusive Physiological Monitoring for Affective Sensing of Computer Users , 2008 .

[21]  Nilanjan Sarkar,et al.  Online stress detection using psychophysiological signals for implicit human-robot cooperation , 2002, Robotica.

[22]  P. Lang.  The emotion probe. Studies of motivation and attention. , 1995, The American psychologist.

[23]  Jon Trinder,et al.  The Humane Interface: New Directions for Designing Interactive Systems , 2002, Interact. Learn. Environ..

[24]  M. Washburn,et al.  Bodily Changes in Pain, Hunger, Fear, and Rage. , 1917 .

[25]  Peter F. Driessen,et al.  Gesture-Based Affective Computing on Motion Capture Data , 2005, ACII.

[26]  Claude Frasson,et al.  Predicting Learners' Emotional Response in Intelligent Distance Learning Systems , 2006, FLAIRS.

[27]  B. Wallin,et al.  Arteriovenous anastomoses and the thermoregulatory shift between cutaneous vasoconstrictor and vasodilator reflexes. , 1995, Journal of the autonomic nervous system.

[28]  J. Lacey.  Psychophysiological approaches to the evaluation of psychotherapeutic process and outcome. , 1959 .

[29]  John J. Magee,et al.  Categorical perception of facial expressions , 1992, Cognition.

[30]  N. Graham,et al.  Stress and acute respiratory infection. , 1986, American journal of epidemiology.

[31]  Andrea Kleinsmith,et al.  Towards Unsupervised Detection of Affective Body Posture Nuances , 2005, ACII.

[32]  Claude Frasson,et al.  The Emotional Conditions of Learning , 2005, FLAIRS.

[33]  Michael E. Dawson,et al.  Emotions and bodily responses : a psychophysiological approach , 1978 .

[34]  H. Lüdtke,et al.  Mathematical procedures in data recording and processing of pupillary fatigue waves , 1998, Vision Research.

[35]  Federico Girosi,et al.  Training support vector machines: an application to face detection , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[36]  M. Dawson,et al.  The electrodermal system , 2007 .

[37]  Veikko Surakka,et al.  Pupil size variation as an indication of affective processing , 2003, Int. J. Hum. Comput. Stud..

[38]  Malek Adjouadi,et al.  Multidimensional Pattern Recognition and Classification of White Blood Cells Using Support Vector Machines , 2005 .

[39]  J. Beatty,et al.  The pupillary system. , 2000 .

[40]  L. Becker,et al.  Family functioning and stress as predictors of influenza B infection. , 1989, The Journal of family practice.

[41]  Nick Campbell,et al.  Perception of affect in speech - towards an automatic processing of paralinguistic information in spoken conversation , 2004, INTERSPEECH.

[42]  C. Darwin.  The Expression of the Emotions in Man and Animals , .

[43]  J. Blondin,et al.  The stress of Stroop performance: physiological and emotional responses to color-word interference, task pacing, and pacing speed. , 1997, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[44]  Lawrence Stark,et al.  Stability, Oscillations, and Noise in the Human Pupil Servomechanism , 1959, Proceedings of the IRE.

[45]  Bernhard Schölkopf,et al.  Comparing support vector machines with Gaussian kernels to radial basis function classifiers , 1997, IEEE Trans. Signal Process..

[46]  Roberto Brunelli,et al.  Person identification using multiple cues , 1995, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Chester W. Darrow,et al.  The galvanic skin reflex (sweating) and blood-pressure as preparatory and facilitative functions. , 1936 .

[48]  L. Berkman.  The role of social relations in health promotion. , 1995, Psychosomatic medicine.

[49]  S. Steinhauer,et al.  Sympathetic and parasympathetic innervation of pupillary dilation during sustained processing. , 2004, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[50]  J. Cacioppo,et al.  Inferring psychological significance from physiological signals. , 1990, The American psychologist.

[51]  Thorsten Joachims,et al.  Making large scale SVM learning practical , 1998 .

[52]  Samy Bengio,et al.  SVMTorch: Support Vector Machines for Large-Scale Regression Problems , 2001, J. Mach. Learn. Res..