Automated Measurement of Facial Expression in Infant-Mother Interaction: A Pilot Study.

Automated facial measurement using computer vision has the potential to objectively document continuous changes in behavior. To examine emotional expression and communication, we used automated measurements to quantify smile strength, eye constriction, and mouth opening in two six-month-old/mother dyads who each engaged in a face-to-face interaction. Automated measurements showed high associations with anatomically based manual coding (concurrent validity); measurements of smiling showed high associations with mean ratings of positive emotion made by naive observers (construct validity). For both infants and mothers, smile strength and eye constriction (the Duchenne marker) were correlated over time, creating a continuous index of smile intensity. Infant and mother smile activity exhibited changing (nonstationary) local patterns of association, suggesting the dyadic repair and dissolution of states of affective synchrony. The study provides insights into the potential and limitations of automated measurement of facial action.

[1]  J. Movellan,et al.  Are Your Eyes Smiling? Detecting genuine smiles with support vector machines and Gabor wavelets , 2001 .

[2]  Takeo Kanade,et al.  Recognizing Action Units for Facial Expression Analysis , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  D. Stern.  The Interpersonal World of the Infant: A View from Psychoanalysis and Developmental Psychology , 2019 .

[4]  Jing Xiao,et al.  Automatic recognition of eye blinking in spontaneously occurring behavior , 2002, Object recognition supported by user interaction for service robots.

[5]  J. Cohn,et al.  Automated face analysis by feature point tracking has high concurrent validity with manual FACS coding. , 1999, Psychophysiology.

[6]  P. Yoder,et al.  Interpreting kappa in observational research: baserate matters. , 2006, American journal of mental retardation : AJMR.

[7]  S. Boker,et al.  Windowed cross-correlation and peak picking for the analysis of variability in the association between behavioral time series. , 2002, Psychological methods.

[8]  J. Cohn,et al.  Infant-mother face-to-face interaction: age and gender differences in coordination and the occurrence of miscoordination. , 1989, Child development.

[9]  A. Fogel,et al.  The temporal structure of face-to-face communication between mothers and infants. , 1980 .

[10]  Gwen Littlewort,et al.  Recognizing facial expression: machine learning and application to spontaneous behavior , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[11]  Yang Wang,et al.  Non-rigid face tracking with enforced convexity and local appearance consistency constraint , 2010, Image Vis. Comput..

[12]  A. Schore.  Affect Regulation and the Origin of the Self: The Neurobiology of Emotional Development , 1994 .

[13]  J. Cohn,et al.  Gender differences in emotional expressivity and self-regulation during early infancy. , 1999, Developmental psychology.

[14]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[15]  P. Ekman,et al.  The Duchenne smile: emotional expression and brain physiology. II. , 1990, Journal of personality and social psychology.

[16]  L. V. Van Egeren,et al.  Mother-infant responsiveness: timing, mutual regulation, and interactional context. , 2001, Developmental psychology.

[17]  Daniel N. Stern,et al.  Mother and infant at play: The dyadic interaction involving facial, vocal, and gaze behaviors. , 1974 .

[18]  Ralph Gross,et al.  Active appearance models with occlusion , 2006, Image Vis. Comput..

[19]  D. Keltner,et al.  PERSONALITY PROCESSES AND INDIVIDUAL DIFFERENCES Expressions of Positive Emotion in Women ' s College Yearbook Pictures and Their Relationship to Personality and Life Outcomes Across Adulthood , 2004 .

[20]  J. Cohn,et al.  Mother–infant face-to-face interaction: The sequence of dyadic states at 3, 6, and 9 months. , 1987 .

[21]  Yanxi Liu,et al.  The role of structural facial asymmetry in asymmetry of peak facial expressions , 2006, Laterality.

[22]  Carlos D. Castillo,et al.  Facial Action Coding , 2009, Encyclopedia of Biometrics.

[23]  K. Schneider,et al.  Preschoolers' attention and emotion in an achievement and an effect game: A longitudinal study , 1992 .

[24]  N. Fox,et al.  Patterns of brain electrical activity during facial signs of emotion in 10-month-old infants. , 1988 .

[25]  Ruth Feldman,et al.  On the origins of background emotions: from affect synchrony to symbolic expression. , 2007, Emotion.

[26]  J. M. Carroll,et al.  Three facial expressions mothers direct to their infants , 2003 .

[27]  Jing Xiao,et al.  Multimodal coordination of facial action, head rotation, and eye motion during spontaneous smiles , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[28]  Howard F. Andrews,et al.  Six-week postpartum maternal self-criticism and dependency and 4-month mother-infant self- and interactive contingencies. , 2007, Developmental psychology.

[29]  H Gu,et al.  The effects of averaging subjective probability estimates between and within judges. , 2000, Journal of experimental psychology. Applied.

[30]  D. Messinger.  Positive and Negative: Infant Facial Expressions and Emotions , 2002 .

[31]  Mark Sheskin,et al.  Visual disengagement in the infant siblings of children with an autism spectrum disorder (ASD) , 2008, Autism : the international journal of research and practice.

[32]  J. Cohn,et al.  Impact of depression on response to comedy: a dynamic facial coding analysis. , 2007, Journal of abnormal psychology.

[33]  L. Wright,et al.  Cocaine exposure is associated with subtle compromises of infants' and mothers' social-emotional behavior and dyadic features of their interaction in the face-to-face still-face paradigm. , 2005, Developmental psychology.

[34]  Roger Bakeman,et al.  Observing Interaction: An Introduction to Sequential Analysis , 1986 .

[35]  E. Tronick.  Emotions and emotional communication in infants. , 1989, The American psychologist.

[36]  J. Cohn,et al.  Use of Automated Facial Image Analysis for Measurement of Emotion Expression , 2004 .

[37]  G. Kochanska,et al.  Mother-child mutually responsive orientation and conscience development: from toddler to early school age. , 2000, Child development.

[38]  P. Ekman,et al.  Felt, false, and miserable smiles , 1982 .

[39]  Jason M. Saragih,et al.  Non-rigid Face Tracking with Local Appearance Consistency Constraint. , 2008, Image and vision computing.

[40]  Tsuhan Chen,et al.  The painful face - Pain expression recognition using active appearance models , 2009, Image Vis. Comput..

[41]  G. Kochanska.  Mutually Responsive Orientation Between Mothers and Their Young Children: A Context for the Early Development of Conscience , 2002 .

[42]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[43]  Gwen Littlewort,et al.  Fully Automatic Facial Action Recognition in Spontaneous Behavior , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[44]  J. Cohn,et al.  Movement Differences between Deliberate and Spontaneous Facial Expressions: Zygomaticus Major Action in Smiling , 2006, Journal of nonverbal behavior.

[45]  J. Cohn,et al.  Infant Smiling Dynamics and Perceived Positive Emotion , 2008, Journal of nonverbal behavior.

[46]  Alyson F. Shapiro,et al.  Effects of normal and perturbed social play on the duration and amplitude of different types of infant smiles. , 2006, Developmental psychology.

[47]  R. Feldman,et al.  Mother-infant affect synchrony as an antecedent of the emergence of self-control. , 1999, Developmental psychology.

[48]  J. Cohn,et al.  Adult Perceptions of Positive and Negative Infant Emotional Expressions , 2005 .

[49]  G. Kochanska,et al.  Implications of the mother-child relationship in infancy for socialization in the second year of life , 1999 .

[50]  Willibald Ruch,et al.  Exhilaration and humor. , 1993 .

[51]  Jeffrey F. Cohn,et al.  Automated Facial Image Analysis: Detecting Improvement in Abnormal Facial Movement After Treatment With Botulinum Toxin A , 2007, Annals of plastic surgery.

[52]  L. Gerstman,et al.  A Method of Defining 'Packages' of Maternal Stimulation and their Functional Significance for the Infant with Mother and Stranger , 1984 .

[53]  P. Ekman,et al.  Smiles when lying. , 1988, Journal of personality and social psychology.

[54]  D. Messinger,et al.  The interactive development of social smiling. , 2007, Advances in child development and behavior.

[55]  P. Ekman,et al.  Behavioral markers and recognizability of the smile of enjoyment. , 1993, Journal of personality and social psychology.

[56]  Xiao-Li Meng,et al.  Comparing correlated correlation coefficients , 1992 .

[57]  Gwen Littlewort,et al.  Automatic Recognition of Facial Actions in Spontaneous Expressions , 2006, J. Multim..

[58]  John J. B. Allen,et al.  The handbook of emotion elicitation and assessment , 2007 .

[59]  P. Ekman,et al.  Behavioral Markers and Recognizability of the Smile of Enjoyment , 1993 .

[60]  B. Lester,et al.  The rhythmic structure of mother-infant interaction in term and preterm infants. , 1985, Child development.

[61]  R. Feldman.  From biological rhythms to social rhythms: Physiological precursors of mother-infant synchrony. , 2006, Developmental psychology.

[62]  D. Messinger,et al.  All smiles are positive, but some smiles are more positive than others. , 2001, Developmental psychology.

[63]  C. Harris.  The Mystery of Ticklish Laughter , 1999, American Scientist.

[64]  Beat Fasel,et al.  Automatic facial expression analysis: a survey , 2003, Pattern Recognit..

[65]  Takeo Kanade,et al.  Evaluation of Gabor-wavelet-based facial action unit recognition in image sequences of increasing complexity , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[66]  J. Cohn,et al.  Mother infant face to face interaction influence is bidirectional and unrelated to periodic cycles in either partner's behavior , 1988 .

[67]  Jeff G. Schneider,et al.  Automatic construction of active appearance models as an image coding problem , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.