Optical computer recognition of facial expressions associated with stress induced by performance demands.

Application of computer vision to track changes in human facial expressions during long-duration spaceflight may be a useful way to unobtrusively detect the presence of stress during critical operations. To develop such an approach, we applied optical computer recognition (OCR) algorithms for detecting facial changes during performance while people experienced both low- and high-stressor performance demands. Workload and social feedback were used to vary performance stress in 60 healthy adults (29 men, 31 women; mean age 30 yr). High-stressor scenarios involved more difficult performance tasks, negative social feedback, and greater time pressure relative to low workload scenarios. Stress reactions were tracked using self-report ratings, salivary cortisol, and heart rate. Subjects also completed personality, mood, and alexithymia questionnaires. To bootstrap development of the OCR algorithm, we had a human observer, blind to stressor condition, identify the expressive elements of the face of people undergoing high- vs. low-stressor performance. Different sets of videos of subjects' faces during performance conditions were used for OCR algorithm training. Subjective ratings of stress, task difficulty, effort required, frustration, and negative mood were significantly increased during high-stressor performance bouts relative to low-stressor bouts (all p < 0.01). The OCR algorithm was refined to provide robust 3-d tracking of facial expressions during head movement. Movements of eyebrows and asymmetries in the mouth were extracted. These parameters are being used in a Hidden Markov model to identify high- and low-stressor conditions. Preliminary results suggest that an OCR algorithm using mouth and eyebrow regions has the potential to discriminate high- from low-stressor performance bouts in 75-88% of subjects. The validity of the workload paradigm to induce differential levels of stress in facial expressions was established. The paradigm also provided the basic stress-related facial expressions required to establish a prototypical OCR algorithm to detect such changes. Efforts are underway to further improve the OCR algorithm by adding facial touching and automating application of the deformable masks and OCR algorithms to video footage of the moving faces as a prelude to blind validation of the automated approach.

[1]  G J Taylor,et al.  Toward the development of a new self-report alexithymia scale. , 1985, Psychotherapy and psychosomatics.

[2]  T. Sejnowski,et al.  Measuring facial expressions by computer image analysis. , 1999, Psychophysiology.

[3]  Sheldon Cohen.  Perceived stress in a probability sample of the United States , 1988 .

[4]  Jiro Gyoba,et al.  Specific brain processing of facial expressions in people with alexithymia: an H2 15O-PET study. , 2003, Brain : a journal of neurology.

[5]  J. Rotter.  Generalized expectancies for internal versus external control of reinforcement. , 1966, Psychological monographs.

[6]  D R Thorne,et al.  The Walter Reed performance assessment battery. , 1985, Neurobehavioral toxicology and teratology.

[7]  K. Scherer,et al.  Handbook of affective sciences. , 2003 .

[8]  N Kanas,et al.  Social and cultural issues during Shuttle/Mir space missions. , 2000, Acta astronautica.

[9]  S R Ellis,et al.  Collision in Space , 2000, Ergonomics in design : the magazine of human factors applications.

[10]  Karen L. Schmidt,et al.  Signal characteristics of spontaneous facial expressions: automatic movement in solitary and social smiles , 2003, Biological Psychology.

[11]  P. Ekman,et al.  The Duchenne smile: emotional expression and brain physiology. II. , 1990, Journal of personality and social psychology.

[12]  Matthew Stone,et al.  An anthropometric face model using variational techniques , 1998, SIGGRAPH.

[13]  Ioannis A. Kakadiaris,et al.  3D human body model acquisition from multiple views , 1995, Proceedings of IEEE International Conference on Computer Vision.

[14]  Bryan Burrough,et al.  Dragonfly: NASA and the Crisis Aboard Mir , 1999 .

[15]  David F. Dinges,et al.  Microcomputer analyses of performance on a portable, simple visual RT task during sustained operations , 1985 .

[16]  J. Aggarwal,et al.  Lower limb kinematics of human walking with the medial axis transformation , 1994, Proceedings of 1994 IEEE Workshop on Motion of Non-rigid and Articulated Objects.

[17]  J. Ridley.  Studies of Interference in Serial Verbal Reactions , 2001 .

[18]  Dimitris N. Metaxas,et al.  Parallel hidden Markov models for American sign language recognition , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[19]  R. Davidson,et al.  Manipulating affective state using extended picture presentations. , 1997, Psychophysiology.

[20]  Ioannis A. Kakadiaris,et al.  Model-based estimation of 3D human motion with occlusion based on active multi-viewpoint selection , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[21]  M. Lorr,et al.  Manual for the Profile of Mood States , 1971 .

[22]  A. Beck,et al.  Internal consistencies of the original and revised Beck Depression Inventory. , 1984, Journal of clinical psychology.

[23]  C. Carver,et al.  Assessing coping strategies: a theoretically based approach. , 1989, Journal of personality and social psychology.

[24]  Ioannis A. Kakadiaris,et al.  Inferring 2D Object Structure from the Deformation of Apparent Contours , 1997, Comput. Vis. Image Underst..

[25]  Larry S. Davis,et al.  Recognizing Human Facial Expressions From Long Image Sequences Using Optical Flow , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Dimitris N. Metaxas,et al.  ASL recognition based on a coupling between HMMs and 3D motion analysis , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[27]  Sven J. Dickinson,et al.  Improving the scope of deformable model shape and motion estimation , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[28]  P. Ekman.  Facial expressions of emotion: an old controversy and new findings. , 1992, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[29]  C. Carver,et al.  Optimism, coping, and health: assessment and implications of generalized outcome expectancies. , 2009, Health psychology : official journal of the Division of Health Psychology, American Psychological Association.

[30]  Daniel J Buysse,et al.  The Pittsburgh sleep quality index: A new instrument for psychiatric practice and research , 1989, Psychiatry Research.

[31]  Timothy F. Cootes,et al.  Automatic Interpretation and Coding of Face Images Using Flexible Models , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  Alex Pentland,et al.  Facial expression recognition using a dynamic model and motion energy , 1995, Proceedings of IEEE International Conference on Computer Vision.

[33]  Dimitris N. Metaxas,et al.  The integration of optical flow and deformable models with applications to human face shape and motion estimation , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[34]  Alex Pentland,et al.  Coding, Analysis, Interpretation, and Recognition of Facial Expressions , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[35]  Theodore Millon,et al.  A theoretical derivation of pathological personalities. , 1986 .

[36]  Dimitris N. Metaxas,et al.  Combining information using hard constraints , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[37]  S. Simpson.  Staying sane in space. , 2000, Scientific American.

[38]  Dimitris N. Metaxas,et al.  Deformable model-based shape and motion analysis from images using motion residual error , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[39]  T J Meyer,et al.  Development and validation of the Penn State Worry Questionnaire. , 1990, Behaviour research and therapy.

[40]  Dimitris N. Metaxas,et al.  Optical Flow Constraints on Deformable Models with Applications to Face Tracking , 2000, International Journal of Computer Vision.

[41]  Jitendra Malik,et al.  Tracking people with twists and exponential maps , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[42]  K. Rohr.  Towards model-based recognition of human movements in image sequences , 1994 .

[43]  Timothy F. Elsmore,et al.  SYNWORK1: A PC-based tool for assessment of performance in a simulated work environment , 1994 .

[44]  N Kanas,et al.  Psychiatric issues affecting long duration space missions. , 1998, Aviation, space, and environmental medicine.

[45]  D. Marlowe,et al.  A new scale of social desirability independent of psychopathology. , 1960, Journal of consulting psychology.

[46]  Pascal Fua,et al.  Object-centered surface reconstruction: Combining multi-image stereo and shading , 1995, International Journal of Computer Vision.

[47]  T. Shallice,et al.  Response suppression, initiation and strategy use following frontal lobe lesions , 1996, Neuropsychologia.

[48]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[49]  Pertti Roivainen,et al.  3-D Motion Estimation in Model-Based Facial Image Coding , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[50]  Pascal Fua,et al.  Variable albedo surface reconstruction from stereo and shape from shading , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[51]  Lynn Stockley,et al.  Specific , 1988 .

[52]  D. Dinges,et al.  Performing while sleepy: Effects of experimentally-induced sleepiness. , 1991 .

[53]  Timothy F. Cootes,et al.  A unified approach to coding and interpreting face images , 1995, Proceedings of IEEE International Conference on Computer Vision.

[54]  Takeo Kanade,et al.  Visual Tracking of High DOF Articulated Structures: an Application to Human Hand Tracking , 1994, ECCV.

[55]  Alex Pentland,et al.  Parametrized structure from motion for 3D adaptive feedback tracking of faces , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[56]  Alan D. Baddeley,et al.  A 3 min reasoning test based on grammatical transformation , 1968 .

[57]  J. Horne,et al.  A self-assessment questionnaire to determine morningness-eveningness in human circadian rhythms. , 1976, International journal of chronobiology.

[58]  Siome Goldenstein,et al.  Statistical Cue Integration in DAG Deformable Models , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[59]  Dimitris N. Metaxas,et al.  Toward Scalability in ASL Recognition: Breaking Down Signs into Phonemes , 1999, Gesture Workshop.

[60]  M. Bradley,et al.  Neuroanatomical correlates of pleasant and unpleasant emotion , 1997, Neuropsychologia.

[61]  Jennifer S. Beer,et al.  Facial expression of emotion. , 2003 .

[62]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[63]  C. Spielberger,et al.  Manual for the State-Trait Anxiety Inventory , 1970 .

[64]  Larry S. Davis,et al.  Towards 3-D model-based tracking and recognition of human movement: a multi-view approach , 1995 .

[65]  Karl J. Friston,et al.  Neuroanatomical correlates of externally and internally generated human emotion. , 1997, The American journal of psychiatry.

[66]  Takeo Kanade,et al.  Model-based tracking of self-occluding articulated objects , 1995, Proceedings of IEEE International Conference on Computer Vision.

[67]  Dimitris N. Metaxas,et al.  Adapting hidden Markov models for ASL recognition by using three-dimensional computer vision methods , 1997, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.