Human Computing and Machine Understanding of Human Behavior: A Survey

A widely accepted prediction is that computing will move to the background, weaving itself into the fabric of our everyday living spaces and projecting the human user into the foreground. If this prediction is to come true, then next generation computing should be about anticipatory user interfaces that should be human-centered, built for humans based on human models. They should transcend the traditional keyboard and mouse to include natural, humanlike interactive functions including understanding and emulating certain human behaviors such as affecti0ve and social signaling. This article discusses how far are we from enabling computers to understand human behavior.

[1]  David H. Evans,et al.  Detection of cough signals in continuous audio recordings using hidden Markov models , 2006, IEEE Transactions on Biomedical Engineering.

[2]  Jeffrey F. Cohn,et al.  Foundations of human computing: facial expression and emotion , 2006, ICMI '06.

[3]  P. Ekman,et al.  The Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and Coding , 1969 .

[4]  Li Deng,et al.  Challenges in adopting speech recognition , 2004, CACM.

[5]  Cheng-Chin Chiang,et al.  A robust method for detecting arbitrarily tilted human faces in color images , 2005, Pattern Recognit. Lett..

[6]  M. Pitt,et al.  Filtering via Simulation: Auxiliary Particle Filters , 1999 .

[7]  Nikos Paragios,et al.  Trajectory-Based Representation of Human Actions , 2007, Artifical Intelligence for Human Computing.

[8]  A F Bobick,et al.  Movement, activity and action: the role of knowledge in the perception of motion. , 1997, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[9]  A. J. Fridlund.  The psychology of facial expression: The new ethology of human facial expressions , 1997 .

[10]  Josef Kittler,et al.  Introduction to the Special Issue on Biometrics: Progress and Directions , 2007, IEEE Trans. Pattern Anal. Mach. Intell..

[11]  Sharon Oviatt,et al.  User-centered modeling and evaluation of multimodal interfaces , 2003, Proc. IEEE.

[12]  Maja Pantic,et al.  Biologically vs. Logic Inspired Encoding of Facial Actions and Emotions in Video , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[13]  L. Rothkrantz,et al.  Toward an affect-sensitive multimodal human-computer interaction , 2003, Proc. IEEE.

[14]  Christine L. Lisetti,et al.  Automatic facial expression interpretation: Where human-computer interaction, artificial intelligence and cognitive science intersect , 2000 .

[15]  Hilary Buxton,et al.  Learning and understanding dynamic scene activity: a review , 2003, Image Vis. Comput..

[16]  Chun Chen,et al.  Audio-visual based emotion recognition - a new approach , 2004, CVPR 2004.

[17]  Sameer Singh,et al.  Video analysis of human dynamics - a survey , 2003, Real Time Imaging.

[18]  Alex Pentland.  Socially Aware Computation and Communication , 2005, Computer.

[19]  David A. van Leeuwen,et al.  Automatic detection of laughter , 2005, INTERSPEECH.

[20]  Donald A. Norman,et al.  Human-centered design considered harmful , 2005, INTR.

[21]  Patrick J. Flynn,et al.  A survey of approaches and challenges in 3D and multi-modal 3D + 2D face recognition , 2006, Comput. Vis. Image Underst..

[22]  Maja Pantic,et al.  Web-based database for facial expression analysis , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[23]  Sandeep Kumar,et al.  Continuous Verification Using Multimodal Biometrics , 2007, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Maja Pantic,et al.  Gaze-X: adaptive affective multimodal interface for single-user office scenarios , 2006, ICMI '06.

[25]  Anton Nijholt,et al.  The virtuality continuum revisited , 2005, CHI EA '05.

[26]  Arun Ross,et al.  Multibiometric systems , 2004, CACM.

[27]  Richard B. Reilly,et al.  Feature analysis for automatic speechreading , 2001, 2001 IEEE Fourth Workshop on Multimedia Signal Processing (Cat. No.01TH8564).

[28]  Takeo Kanade,et al.  Shape-from-silhouette of articulated objects and its use for human body kinematics estimation and motion capture , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[29]  Peter Robinson,et al.  Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[30]  Susan T. Dumais,et al.  The vocabulary problem in human-system communication , 1987, CACM.

[31]  M. Bartlett,et al.  Machine Analysis of Facial Expressions , 2007 .

[32]  Hatice Gunes,et al.  Affect recognition from face and body: early fusion vs. late fusion , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[33]  M. Heller,et al.  Depression and Suicide Faces , 2005 .

[34]  Yuxiao Hu,et al.  Audio-visual emotion recognition in adult attachment interview , 2006, ICMI '06.

[35]  Mark C. Coulson.  Attributing Emotion to Static Body Postures: Recognition Accuracy, Confusions, and Viewpoint Dependence , 2004 .

[36]  Gwen Littlewort,et al.  Fully Automatic Facial Action Recognition in Spontaneous Behavior , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[37]  Narendra Ahuja,et al.  Detecting Faces in Images: A Survey , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[38]  Jean-Marc Odobez,et al.  A probabilistic framework for joint head tracking and pose estimation , 2004, ICPR 2004.

[39]  Gregory D. Abowd,et al.  A Conceptual Framework and a Toolkit for Supporting the Rapid Prototyping of Context-Aware Applications , 2001, Hum. Comput. Interact..

[40]  Andrew T Duchowski,et al.  A breadth-first survey of eye-tracking applications , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[41]  Harriet J. Nock,et al.  Multimodal processing by finding common cause , 2004, CACM.

[42]  Jennifer S. Beer,et al.  Facial expression of emotion. , 2003 .

[43]  J. Cacioppo,et al.  The psychophysiology of emotion. , 1993 .

[44]  Thomas Rist,et al.  Lost in ambient intelligence? , 2004, CHI EA '04.

[45]  Anton Nijholt,et al.  Human computing, virtual humans and artificial imperfection , 2006, ICMI '06.

[46]  Maja Pantic,et al.  Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[47]  A. Brodal,et al.  Neurological Anatomy in Relation to Clinical Medicine , 1950 .

[48]  Samy Bengio,et al.  Detecting group interest-level in meetings , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[49]  Michael Isard,et al.  CONDENSATION—Conditional Density Propagation for Visual Tracking , 1998, International Journal of Computer Vision.

[50]  Maja Pantic,et al.  Particle filtering with factorized likelihoods for tracking facial features , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[51]  Dirk Heylen,et al.  Social User Interfaces , 2006 .

[52]  Heinrich H. Bülthoff,et al.  The components of conversational facial expressions , 2004, APGV '04.

[53]  Mohammed Yeasin,et al.  Speech-gesture driven multimodal interfaces for crisis management , 2003, Proc. IEEE.

[54]  Ananth N. Iyer,et al.  Emotion Detection From Infant Facial Expressions And Cries , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[55]  M. Weiser.  The Computer for the Twenty-First Century , 1991 .

[56]  Jing Xiao,et al.  Automatic analysis and recognition of brow actions and head motion in spontaneous facial behavior , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[57]  Emile H. L. Aarts.  Ambient intelligence drives open innovation , 2005, INTR.

[58]  Jeffrey F. Cohn,et al.  The Timing of Facial Motion in posed and Spontaneous Smiles , 2003, Int. J. Wavelets Multiresolution Inf. Process..

[59]  Manuele Bicego,et al.  Unsupervised scene analysis: a hidden Markov model approach , 2006 .

[60]  Thomas S. Huang,et al.  Connected vibrations: a modal analysis approach for non-rigid motion tracking , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[61]  R. E. Kalman,et al.  A New Approach to Linear Filtering and Prediction Problems , 2002 .

[62]  Loïc Kessous,et al.  Modeling Naturalistic Affective States Via Facial, Vocal, and Bodily Expressions Recognition , 2007, Artifical Intelligence for Human Computing.

[63]  W. Rinn,et al.  The neuropsychology of facial expression: a review of the neurological and psychological mechanisms for producing facial expressions. , 1984, Psychological bulletin.

[64]  J. Russell,et al.  Facial and vocal expressions of emotion. , 2003, Annual review of psychology.

[65]  Qiang Ji,et al.  Information extraction from image sequences of real-world facial expressions , 2005, Machine Vision and Applications.

[66]  J. Russell,et al.  The psychology of facial expression: Frontmatter , 1997 .

[67]  Seth J. Teller,et al.  Particle Video: Long-Range Motion Estimation Using Point Trajectories , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[68]  Maja Pantic,et al.  Fully Automatic Facial Action Unit Detection and Temporal Analysis , 2006, 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06).

[69]  Simon Haykin,et al.  Special Issue on Sequential State Estimation , 2004, Proc. IEEE.

[70]  Shumin Zhai,et al.  Introduction to sensing-based interaction , 2005, TCHI.

[71]  Azriel Rosenfeld,et al.  Face recognition: A literature survey , 2003, CSUR.

[72]  Mark Weiser.  The computer for the 21st century , 1991 .

[73]  Thad Starner,et al.  The Challenges of Wearable Computing: Part 1 , 2001, IEEE Micro.

[74]  Tieniu Tan,et al.  Recent developments in human motion analysis , 2003, Pattern Recognit..

[75]  P. Ekman.  Darwin, Deception, and Facial Expression , 2003, Annals of the New York Academy of Sciences.

[76]  Alex Pentland,et al.  Modeling Influence Between Experts , 2007, Artifical Intelligence for Human Computing.

[77]  Mohan M. Trivedi,et al.  Robust real-time detection, tracking, and pose estimation of faces in video streams , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[78]  N. Ambady,et al.  Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis. , 1992 .

[79]  C. Izard.  Emotions and facial expressions: A perspective from Differential Emotions Theory. , 1997 .

[80]  Björn Stenger,et al.  Model-based hand tracking using a hierarchical Bayesian filter , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[81]  Amanda C.C. Williams,et al.  Facial expression of pain: An evolutionary account , 2002, Behavioral and Brain Sciences.

[82]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[83]  Maja Pantic,et al.  Spontaneous vs. posed facial behavior: automatic analysis of brow actions , 2006, ICMI '06.

[84]  Marco Costa,et al.  Social Presence, Embarrassment, and Nonverbal Behavior , 2001 .

[85]  Oudeyer Pierre-Yves,et al.  The production and recognition of emotions in speech: features and algorithms , 2003 .

[86]  K. Scherer,et al.  Vocal expression of affect , 2005 .