Using face and object detection to quantify looks during social interactions

Quantifying gaze is important in various realms, such as evaluating atypical social looking behavior in autism spectrum disorder. This paper reports on a system that uses eye-tracking glasses and object/face detection to quantify looks. The algorithms use Viola-Jones face detection with feature point tracking and Faster-RCNN object detection trained for three objects, followed by a runlength algorithm to declare the start and end of looks. Results are presented in terms of bounding box overlap and accuracy of looks compared to a manual ground truth. The system can be useful for quantifying gaze behavior during dynamic social interactions.

[1]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[2]  James M. Rehg,et al.  Detecting eye contact using wearable eye-tracking glasses , 2012, UbiComp.

[3]  Song Han,et al.  Learning both Weights and Connections for Efficient Neural Network , 2015, NIPS.

[4]  Eric Courchesne,et al.  Slowed orienting of covert visual-spatial attention in autism: Specific deficits associated with cerebellar and parietal abnormality , 1996, Development and Psychopathology.

[5]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[6]  R. C. Langford.  How People Look at Pictures, A Study of the Psychology of Perception in Art. , 1936 .

[7]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Masako Hirotani,et al.  Joint attention helps infants learn new words: event-related potential evidence , 2009, Neuroreport.

[9]  A. Billard,et al.  Investigating Gaze of Children with ASD in Naturalistic Settings , 2012, PloS one.

[10]  Joseph H. Goldberg,et al.  Identifying fixations and saccades in eye-tracking protocols , 2000, ETRA.

[11]  James M. Rehg,et al.  Detecting Gaze Towards Eyes in Natural Social Interactions and Its Use in Child Assessment , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[12]  A. Billard,et al.  Social orienting of children with autism to facial expressions and speech: a study with a wearable eye-tracker in naturalistic settings , 2013, Front. Psychol..

[13]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Dare A. Baldwin,et al.  Understanding the link between joint attention and language. , 1995 .

[15]  B. Ingersoll,et al.  Brief Report: High and Low Level Initiations of Joint Attention, and Response to Joint Attention: Differential Relationships with Language and Imitation , 2015, Journal of autism and developmental disorders.

[16]  P. Mundy,et al.  A review of joint attention and social‐cognitive brain systems in typical development and autism spectrum disorder , 2018, The European journal of neuroscience.

[17]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[18]  Jean-Philippe Thiran,et al.  Analysis of Head-Mounted Wireless Camera Videos for Early Diagnosis of Autism , 2008, Computer Recognition Systems 2.