Neural Signals Evoked by Stimuli of Increasing Social Scene Complexity Are Detectable at the Single-Trial Level and Right Lateralized

Classification of neural signals at the single-trial level and the study of their relevance in affective and cognitive neuroscience are still in their infancy. Here we investigated the neurophysiological correlates of conditions of increasing social scene complexity using 3D human models as targets of attention, which may also be important in autism research. Challenging single-trial statistical classification of EEG neural signals was attempted for detection of oddball stimuli with increasing social scene complexity. Stimuli had an oddball structure and were as follows: 1) flashed schematic eyes, 2) simple 3D faces flashed between averted and non-averted gaze (only eye position changing), 3) simple 3D faces flashed between averted and non-averted gaze (head and eye position changing), 4) animated avatar alternated its gaze direction to the left and to the right (head and eye position), 5) environment with 4 animated avatars all of which change gaze and one of which is the target of attention. We found a late (> 300 ms) neurophysiological oddball correlate for all conditions irrespective of their complexity as assessed by repeated measures ANOVA. We attempted single-trial detection of this signal with automatic classifiers and obtained a significant balanced accuracy classification of around 79%, which is noteworthy given the amount of scene complexity. Lateralization analysis showed a specific right lateralization only for more complex realistic social scenes. In sum, complex ecological animations with social content elicit neurophysiological events which can be characterized even at the single-trial level. These signals are right lateralized. These finding paves the way for neuroscientific studies in affective neuroscience based on complex social scenes, and given the detectability at the single trial level this suggests the feasibility of brain computer interfaces that can be applied to social cognition disorders such as autism.

[1]  N. Sadato,et al.  Attention to emotion modulates fMRI activity in human right superior temporal sulcus. , 2001, Brain research. Cognitive brain research.

[2]  Johan D. Carlin,et al.  The neural basis of eye gaze processing , 2013, Current Opinion in Neurobiology.

[3]  Deborah A. Hall,et al.  Reading Fluent Speech from Talking Faces: Typical Brain Networks and Individual Differences , 2005, Journal of Cognitive Neuroscience.

[4]  Johan D. Carlin,et al.  Direction-Sensitive Codes for Observed Head Turns in Human Superior Temporal Sulcus , 2011, Cerebral cortex.

[5]  Robert T. Knight,et al.  Superior Temporal SulcusIt's My Area: Or Is It? , 2008, Journal of Cognitive Neuroscience.

[6]  M. Castelo‐Branco,et al.  Functional parcellation of the operculo-insular cortex in perceptual decision making: An fMRI study , 2012, Neuropsychologia.

[7]  Kai Kaspar What Guides Visual Overt Attention under Natural Conditions? Past and Future Research , 2013, ISRN neuroscience.

[8]  H. Demaree,et al.  Brain lateralization of emotional processing: historical roots and a future incorporating "dominance". , 2005, Behavioral and cognitive neuroscience reviews.

[9]  G. McCarthy,et al.  Dynamic perception of facial affect and identity in the human brain. , 2003, Cerebral Cortex.

[10]  Urbano Nunes,et al.  Statistical spatial filtering for a P300-based BCI: Tests in able-bodied, and patients with cerebral palsy and amyotrophic lateral sclerosis , 2011, Journal of Neuroscience Methods.

[11]  Francesco Piccione,et al.  Task and Stimulation Paradigm Effects in a P300 Brain Computer Interface Exploitable in a Virtual Environment: A Pilot Study , 2008, PsychNology J..

[12]  Zoran Nenadic,et al.  Pushing the Communication Speed Limit of a Noninvasive BCI Speller , 2012, ArXiv.

[13]  Febo Cincotti,et al.  Out of the frying pan into the fire--the P300-based BCI faces real-world challenges. , 2011, Progress in brain research.

[14]  A. Calder,et al.  Neural mechanisms of social attention , 2009, Trends in Cognitive Sciences.

[15]  J. Haxby,et al.  The distributed human neural system for face perception , 2000, Trends in Cognitive Sciences.

[16]  H. Semlitsch,et al.  P300 event-related potentials and cognitive function in social phobia , 2004, Psychiatry Research: Neuroimaging.

[17]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[18]  F Rösler,et al.  A correction method for DC drift artifacts. , 1993, Electroencephalography and clinical neurophysiology.

[19]  Peter Thier,et al.  Cortical processing of head- and eye-gaze cues guiding joint social attention , 2011, NeuroImage.

[20]  P. MacNeilage Divided Brains: the Biology and Behaviour of Brain Asymmetries Leslie J. Rogers Giorgio Vallortigara , 2013, Animal Behaviour.

[21]  Jonathan R. Wolpaw Brain-computer interfaces: progress, problems, and possibilities , 2012, IHI '12.

[22]  M. Castelo‐Branco,et al.  Comparison of a row-column speller vs. a novel lateral single-character speller: Assessment of BCI for severe motor disabled patients , 2012, Clinical Neurophysiology.

[23]  S. Baron-Cohen,et al.  Gaze Perception Triggers Reflexive Visuospatial Orienting , 1999 .

[24]  G. McCarthy,et al.  When Strangers Pass , 2004, Psychological science.

[25]  M. Siniscalchi Divided brains. The biology and behaviour of brain asymmetries , 2014 .

[26]  Alan Kingstone,et al.  Covert and overt orienting to gaze direction cues and the effects of fixation offset , 2003, Neuroreport.

[27]  Gemma C. Garriga,et al.  Permutation Tests for Studying Classifier Performance , 2009, 2009 Ninth IEEE International Conference on Data Mining.

[28]  C. C. Duncan,et al.  Event-related potentials in clinical research: Guidelines for eliciting, recording, and quantifying mismatch negativity, P300, and N400 , 2009, Clinical Neurophysiology.

[29]  A. Saygin Superior temporal and premotor brain areas necessary for biological motion perception. , 2007, Brain : a journal of neurology.

[30]  Mikko Sams,et al.  Naturalistic fMRI Mapping Reveals Superior Temporal Sulcus as the Hub for the Distributed Brain Network for Social Perception , 2012, Front. Hum. Neurosci..

[31]  Salil H. Patel,et al.  Characterization of N200 and P300: Selected Studies of the Event-Related Potential , 2005, International journal of medical sciences.

[32]  T. Allison,et al.  Temporal Cortex Activation in Humans Viewing Eye and Mouth Movements , 1998, The Journal of Neuroscience.

[33]  David C. Zhu,et al.  The Role of the Right Hemisphere for Processing of Social Interactions in Normal Adults Using Functional Magnetic Resonance Imaging , 2011, Neuropsychobiology.

[34]  G. McCarthy,et al.  Perception of dynamic changes in facial affect and identity in autism. , 2007, Social cognitive and affective neuroscience.

[35]  J. Driver,et al.  A left visual field advantage in perception of gaze direction , 2002, Neuropsychologia.

[36]  R. Sassi In this issue/abstract thinking: mothers, babies, and amygdalae. , 2012, Journal of the American Academy of Child and Adolescent Psychiatry.

[37]  Ana Susac,et al.  Neurodynamic Studies on Emotional and Inverted Faces in an Oddball Paradigm , 2003, Brain Topography.

[38]  A. Klin,et al.  Parsing heterogeneity in autism spectrum disorders: visual scanning of dynamic social scenes in school-aged children. , 2012, Journal of the American Academy of Child and Adolescent Psychiatry.

[39]  J. Polich Updating P300: An integrative theory of P3a and P3b , 2007, Clinical Neurophysiology.

[40]  T. Subba Rao,et al.  Classification, Parameter Estimation and State Estimation: An Engineering Approach Using MATLAB , 2004 .

[41]  Alan C. Evans,et al.  Specific Involvement of Human Parietal Systems and the Amygdala in the Perception of Biological Motion , 1996, The Journal of Neuroscience.

[42]  E. Donchin,et al.  Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. , 1988, Electroencephalography and clinical neurophysiology.

[43]  W. He,et al.  Processing of facial expressions of emotions in healthy volunteers: An exploration with event-related potentials and personality traits , 2012, Neurophysiologie Clinique/Clinical Neurophysiology.

[44]  Miguel Castelo-Branco,et al.  Impaired processing of 3D motion-defined faces in mild cognitive impairment and healthy aging: an fMRI study. , 2013, Cerebral cortex.

[45]  N. Emery,et al.  The eyes have it: the neuroethology, function and evolution of social gaze , 2000, Neuroscience & Biobehavioral Reviews.

[46]  Ad Aertsen,et al.  Review of the BCI Competition IV , 2012, Front. Neurosci..

[47]  R. Malach,et al.  Intersubject Synchronization of Cortical Activity During Natural Vision , 2004, Science.

[48]  M. Crommelinck,et al.  Discrimination of emotional facial expressions in a visual oddball task: an ERP study , 2002, Biological Psychology.

[49]  A. Cichocki,et al.  A novel BCI based on ERP components sensitive to configural processing of human faces , 2012, Journal of neural engineering.

[50]  A. Gunji,et al.  Facial identity recognition in children with autism spectrum disorders revealed by P300 analysis: A preliminary study , 2013, Brain and Development.

[51]  Dana H. Ballard,et al.  Single trial P3 epoch recognition in a virtual environment , 2000, Neurocomputing.

[52]  B. Ingersoll,et al.  The use of innovative computer technology for teaching social communication to individuals with autism spectrum disorders , 2011 .

[53]  Jérémie Mattout,et al.  Brain-Computer Interfaces: A Neuroscience Paradigm of Social Interaction? A Matter of Perspective , 2012, Front. Hum. Neurosci..

[54]  L. Koegel,et al.  Improving Social Initiations in Young Children with Autism Using Reinforcers with Embedded Social Interactions , 2009, Journal of autism and developmental disorders.

[55]  J. Wolpaw,et al.  Brain-Computer Interfaces: Principles and Practice , 2012 .

[56]  A. Brancucci,et al.  Asymmetries of the human social brain in the visual, auditory and chemical modalities , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[57]  R. Blake,et al.  Brain Areas Involved in Perception of Biological Motion , 2000, Journal of Cognitive Neuroscience.

[58]  L. Regolin,et al.  Chicks discriminate human gaze with their right hemisphere , 2007, Behavioural Brain Research.

[59]  M. Castelo‐Branco,et al.  Robust Single Trial Identification of Conscious Percepts Triggered by Sensory Events of Variable Saliency , 2014, PloS one.

[60]  E. Carr,et al.  Multiple Effects of Joint Attention Intervention for Children With Autism , 2006, Behavior modification.

[61]  Christina Whalen,et al.  The Collateral Effects of Joint Attention Training on Social Initiations, Positive Affect, Imitation, and Spontaneous Speech for Young Children with Autism , 2006, Journal of autism and developmental disorders.

[62]  T. Allison,et al.  Social perception from visual cues: role of the STS region , 2000, Trends in Cognitive Sciences.

[63]  L. Jakobson,et al.  Developmental changes in attention to faces and bodies in static and dynamic scenes , 2014, Front. Psychol..

[64]  J. W. Minett,et al.  Optimizing the P300-based brain–computer interface: current status, limitations and future directions , 2011, Journal of neural engineering.

[65]  Toshikazu Hasegawa,et al.  Deviant gaze processing in children with autism: an ERP study , 2005, Neuropsychologia.

[66]  F. Lopera,et al.  Brain potentials reflect residual face processing in a case of prosopagnosia , 2004, Cognitive neuropsychology.

[67]  L. Regolin,et al.  Cerebral and behavioural assymetries in animal social recognition , 2012 .

[68]  J. Schultz,et al.  Natural facial motion enhances cortical responses to faces , 2009, Experimental Brain Research.

[69]  A. Kübler,et al.  Flashing characters with famous faces improves ERP-based brain–computer interface performance , 2011, Journal of neural engineering.

[70]  P. McGuire,et al.  Cortical substrates for the perception of face actions: an fMRI study of the specificity of activation for seen speech and for meaningless lower-face acts (gurning). , 2001, Brain research. Cognitive brain research.

[71]  Ursula Bellugi,et al.  Do extraverts process social stimuli differently from introverts? , 2011, Cognitive neuroscience.

[72]  A. Kingstone Taking a real look at social attention , 2009, Current Opinion in Neurobiology.

[73]  S. Zeki,et al.  Functional brain mapping during free viewing of natural scenes , 2004, Human brain mapping.

[74]  N. George,et al.  When eye creates the contact! ERP evidence for early dissociation between direct and averted gaze motion processing , 2007, Neuropsychologia.

[75]  Gregory McCarthy,et al.  Taking an “intentional stance” on eye-gaze shifts: A functional neuroimaging study of social perception in children , 2005, NeuroImage.