Frontal and occipito-temporal Theta activity as marker of error monitoring in Human-Avatar joint performance

The discrepancy between expected outcome and real outcome is at the base of error coding, a crucial process during adaptive behaviours. Previous studies indicate that performing or observing errors elicits specific EEG markers (e.g. Theta). Here we show how unexpected changes in the movement trajectory of the virtual co-actor in a human-avatar dyadic paradigm are mapped into the error-monitoring system of the human partner. We asked individuals to synchronize their reach-to-grasp movements with those of a virtual partner in conditions that did (Interactive condition), or did not, require (Cued condition) spatio-temporal adaptation to the partner’s actions. Crucially, in certain trials the virtual partner suddenly changed its movement trajectory; thereby violating the human participant’s expectation. These trials showed that fronto-central error-related EEG markers increased as a function of the individual’s reliance on their partner’s behaviour. Source localization showed that observing violations of the expected movements also generated a Theta increase over occipito-temporal regions, highlighting visuo-motor processing during erroneous interactions. Significance Statement Our ability to coordinate with peers relies upon moment-to-moment prediction and integration of visual (i.e. observing the movements of others) and motor (completing our own actions) information. However, when the behaviour of our partners changes unexpectedly, our prediction appears to be incorrect. Here, we describe EEG error-related neuromarkers (ERN/Pe - Theta/Alpha modulations): when human participants perform a joint reach-to-grasp task with a virtual partner. We show that unexpected changes of the avatar trajectory are mapped into EEG error-markers according to the degree of interpersonal interdependence. Moreover, source analysis highlights that fronto-central and occipito-temporal regions generate Theta activity associated with processing visuo-motor information during social interactions.

[1]  E. Wagenmakers A practical solution to the pervasive problems ofp values , 2007, Psychonomic bulletin & review.

[2]  Frank Van Overwalle,et al.  Understanding others' actions and goals by mirror and mentalizing systems: A meta-analysis , 2009, NeuroImage.

[3]  P. Downing,et al.  The lateral occipitotemporal cortex in action , 2015, Trends in Cognitive Sciences.

[4]  Michael X. Cohen,et al.  Error-related medial frontal theta activity predicts cingulate-related structural connectivity , 2011, NeuroImage.

[5]  Robert Oostenveld,et al.  FieldTrip: Open Source Software for Advanced Analysis of MEG, EEG, and Invasive Electrophysiological Data , 2010, Comput. Intell. Neurosci..

[6]  Jonathan D. Cohen,et al.  Anterior Cingulate Cortex, Conflict Monitoring, and Levels of Processing , 2001, NeuroImage.

[7]  C. Urgesi,et al.  The Neural Basis of Body Form and Body Action Agnosia , 2008, Neuron.

[8]  S. T. Grafton,et al.  Pouring or chilling a bottle of wine: an fMRI study on the prospective planning of object-directed actions , 2012, Experimental Brain Research.

[9]  T. Poggio,et al.  Cognitive neuroscience: Neural mechanisms for the recognition of biological movements , 2003, Nature Reviews Neuroscience.

[10]  Erich Schröger,et al.  Digital filter design for electrophysiological data – a practical approach , 2015, Journal of Neuroscience Methods.

[11]  Salvatore Maria Aglioti,et al.  Influence of cognitive stance and physical perspective on subjective and autonomic reactivity to observed pain and pleasure: An immersive virtual reality study , 2019, Consciousness and Cognition.

[12]  T. Sejnowski,et al.  Removal of eye activity artifacts from visual event-related potentials in normal and clinical subjects , 2000, Clinical Neurophysiology.

[13]  Zoltan Dienes,et al.  Using Bayes to get the most out of non-significant results , 2014, Front. Psychol..

[14]  Karl J. Friston,et al.  The mirror-neuron system: a Bayesian perspective. , 2007, Neuroreport.

[15]  Joseph M. Orr,et al.  The Role of the Error Positivity in the Conscious Perception of Errors , 2011, The Journal of Neuroscience.

[16]  H. Bekkering,et al.  Modulation of activity in medial frontal and motor cortices during error observation , 2004, Nature Neuroscience.

[17]  J. Pineda The functional significance of mu rhythms: Translating “seeing” and “hearing” into “doing” , 2005, Brain Research Reviews.

[18]  Cosimo Urgesi,et al.  Magnetic Stimulation of Extrastriate Body Area Impairs Visual Processing of Nonfacial Body Parts , 2004, Current Biology.

[19]  Karl J. Friston,et al.  Predictive coding: an account of the mirror neuron system , 2007, Cognitive Processing.

[20]  Jennifer Wiley,et al.  What Are the Odds? A Practical Guide to Computing and Reporting Bayes Factors , 2014, J. Probl. Solving.

[21]  G. Rizzolatti,et al.  I Know What You Are Doing A Neurophysiological Study , 2001, Neuron.

[22]  Salvatore Maria Aglioti,et al.  Wronger than wrong: Graded mapping of the errors of an avatar in the performance monitoring system of the onlooker , 2018, NeuroImage.

[23]  W. Singer,et al.  Dynamic predictions: Oscillations and synchrony in top–down processing , 2001, Nature Reviews Neuroscience.

[24]  M. Herrmann,et al.  Source localization (LORETA) of the error-related-negativity (ERN/Ne) and positivity (Pe). , 2004, Brain research. Cognitive brain research.

[25]  Kent A. Kiehl,et al.  Neural correlates of the processing of another’s mistakes: A possible underpinning for social and observational learning , 2008, NeuroImage.

[26]  M. Botvinick,et al.  Anterior cingulate cortex, error detection, and the online monitoring of performance. , 1998, Science.

[27]  Matteo Candidi,et al.  Causative role of left aIPS in coding shared goals during human–avatar complementary joint actions , 2015, Nature Communications.

[28]  John J. B. Allen,et al.  Theta EEG dynamics of the error-related negativity , 2007, Clinical Neurophysiology.

[29]  Y. Paulignan,et al.  Neural correlates of non-verbal social interactions: A dual-EEG study , 2014, Neuropsychologia.

[30]  D. Meyer,et al.  A Neural System for Error Detection and Compensation , 1993 .

[31]  Salvatore Maria Aglioti,et al.  Embodying Others in Immersive Virtual Reality: Electro-Cortical Signatures of Monitoring the Errors in the Actions of an Avatar Seen from a First-Person Perspective , 2016, The Journal of Neuroscience.

[32]  Joseph P. McCleery,et al.  EEG evidence for mirror neuron dysfunction in autism spectrum disorders. , 2005, Brain research. Cognitive brain research.

[33]  S. M. Aglioti,et al.  Electrocortical signatures of detecting errors in the actions of others: An EEG study in pianists, non-pianist musicians and musically naïve people , 2016, Neuroscience.

[34]  A. Avenanti,et al.  Action-related dynamic changes in inferior frontal cortex effective connectivity: A TMS/EEG coregistration study , 2018, Cortex.

[35]  M. Candidi,et al.  Kinematics fingerprints of leader and follower role-taking during cooperative joint actions , 2013, Experimental Brain Research.

[36]  F. Varela,et al.  Measuring phase synchrony in brain signals , 1999, Human brain mapping.

[37]  M. Candidi,et al.  Representation of body identity and body actions in extrastriate body area and ventral premotor cortex , 2007, Nature Neuroscience.

[38]  M. Candidi,et al.  Catching on it early: Bodily and brain anticipatory mechanisms for excellence in sport. , 2017, Progress in brain research.

[39]  C. Pantev,et al.  A Beamformer Analysis of MEG Data Reveals Frontal Generators of the Musically Elicited Mismatch Negativity , 2013, PloS one.

[40]  Paul E. Downing,et al.  An event-related potential component sensitive to images of the human body , 2006, NeuroImage.

[41]  J. Kelso,et al.  From the Cover : The phi complex as a neuromarker of human social coordination , 2007 .

[42]  Quentin Moreau,et al.  Dynamics of Social Interaction: Kinematic Analysis of a Joint Action , 2016, Front. Psychol..

[43]  Hermann Haken,et al.  Exploring the Brain , 2013 .

[44]  Salvatore Maria Aglioti,et al.  Transitory Inhibition of the Left Anterior Intraparietal Sulcus Impairs Joint Actions: A Continuous Theta-Burst Stimulation Study , 2018, Journal of Cognitive Neuroscience.

[45]  Gaetano Tieri,et al.  Interactor's body shape does not affect visuo-motor interference effects during motor coordination. , 2019, Acta psychologica.

[46]  L. Parkkonen,et al.  Evidence for a general performance‐monitoring system in the human brain , 2018, Human brain mapping.

[47]  C. Keysers,et al.  μ-Suppression during Action Observation and Execution Correlates with BOLD in Dorsal Premotor, Inferior Parietal, and SI Cortices , 2011, The Journal of Neuroscience.

[48]  R. Hari,et al.  Characterization of spontaneous MEG rhythms in healthy adults. , 1994, Electroencephalography and clinical neurophysiology.

[49]  K. R. Ridderinkhof,et al.  To P(E) or not to P(E): a P3-like ERP component reflecting the processing of response errors. , 2009, Psychophysiology.

[50]  Lucia M. Sacheli,et al.  Social cues to joint actions: the role of shared goals , 2015, Front. Psychol..

[51]  Giovanni Pezzulo,et al.  Studying mirror mechanisms within generative and predictive architectures for joint action , 2013, Cortex.

[52]  F. L. D. Silva,et al.  Event-related EEG/MEG synchronization and desynchronization: basic principles , 1999, Clinical Neurophysiology.

[53]  C. Keysers,et al.  Hebbian learning and predictive mirror neurons for actions, sensations and emotions , 2014, Philosophical Transactions of the Royal Society B: Biological Sciences.

[54]  Clay B. Holroyd,et al.  Evidence for hierarchical error processing in the human brain , 2006, Neuroscience.

[55]  D. V. van Essen,et al.  Computerized Mappings of the Cerebral Cortex: A Multiresolution Flattening Method and a Surface-Based Coordinate System , 1996, Journal of Cognitive Neuroscience.

[56]  K. Yoshizaki,et al.  Priority for one’s own stimulus in joint performance: evidence from an event-related potential study , 2016, Neuroreport.

[57]  M. Candidi,et al.  Autistic traits affect interpersonal motor coordination by modulating strategic use of role-based behavior , 2017, Molecular Autism.

[58]  Thérèse J. M. Overbeek,et al.  Dissociable Components of Error Processing on the Functional Significance of the Pe Vis-à-vis the Ern/ne Performance Monitoring Processes Reflected in the Ne and Pe Review of Studies That Report Both Ne and Pe: Associations and Dissociations Pharmacological Effects , 2022 .

[59]  N. Kanwisher,et al.  The Human Body , 2001 .

[60]  Ricarda I. Schubotz,et al.  An event-related potential study on the observation of erroneous everyday actions , 2007, Cognitive, affective & behavioral neuroscience.

[61]  T. Endrass,et al.  Error Awareness in a Saccade Countermanding Task , 2005 .

[62]  Janeen D. Loehr,et al.  It's not just my fault: Neural correlates of feedback processing in solo and joint action , 2015, Biological Psychology.

[63]  A. Caramazza,et al.  Closely overlapping responses to tools and hands in left lateral occipitotemporal cortex. , 2012, Journal of neurophysiology.

[64]  Salvatore Maria Aglioti,et al.  Somatotopic mapping of piano fingering errors in sensorimotor experts: TMS studies in pianists and visually trained musically naives. , 2014, Cerebral cortex.

[65]  D. Kourtis,et al.  Predictive representation of other people's actions in joint action planning: An EEG study , 2013, Social neuroscience.

[66]  Sidney J Segalowitz,et al.  Development of Response-Monitoring ERPs in 7- to 25-Year-Olds , 2004, Developmental neuropsychology.

[67]  Natalie Sebanz,et al.  Prediction in Joint Action: What, When, and Where , 2009, Top. Cogn. Sci..

[68]  Leonie Koban,et al.  When your errors make me lose or win: Event-related potentials to observed errors of cooperators and competitors , 2010, Social neuroscience.

[69]  J. Rouder Optional stopping: No problem for Bayesians , 2014, Psychonomic bulletin & review.

[70]  What Body Parts Reveal about the Organization of the Brain , 2010, Neuron.

[71]  E. Procyk,et al.  Behavioral Shifts and Action Valuation in the Anterior Cingulate Cortex , 2008, Neuron.

[72]  M. Candidi,et al.  Come together: human–avatar on-line interactions boost joint-action performance in apraxic patients , 2017, Social cognitive and affective neuroscience.

[73]  G. Rizzolatti,et al.  The mirror-neuron system. , 2004, Annual review of neuroscience.

[74]  M. Botvinick,et al.  Conflict monitoring and cognitive control. , 2001, Psychological review.

[75]  Aina Puce,et al.  Electrophysiology and brain imaging of biological motion. , 2003, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[76]  Robert Oostenveld,et al.  Motor-cortical beta oscillations are modulated by correctness of observed action , 2008, NeuroImage.

[77]  C. Urgesi,et al.  Action anticipation and motor resonance in elite basketball players , 2008, Nature Neuroscience.

[78]  M. Candidi,et al.  Inhibition of left anterior intraparietal sulcus shows that mutual adjustment marks dyadic joint-actions in humans , 2018, Social cognitive and affective neuroscience.

[79]  M. Candidi,et al.  And Yet They Act Together: Interpersonal Perception Modulates Visuo-Motor Interference and Mutual Adjustments during a Joint-Grasping Task , 2012, PloS one.

[80]  Stefanie Schuch,et al.  On observing another person’s actions: Influences of observed inhibition and errors , 2007, Perception & psychophysics.

[81]  Joshua W. Brown,et al.  Medial prefrontal cortex as an action-outcome predictor , 2011, Nature Neuroscience.

[82]  A. Engel,et al.  Trial-by-Trial Coupling of Concurrent Electroencephalogram and Functional Magnetic Resonance Imaging Identifies the Dynamics of Performance Monitoring , 2005, The Journal of Neuroscience.

[83]  J. Hohnsbein,et al.  ERP components on reaction errors and their functional significance: a tutorial , 2000, Biological Psychology.

[84]  Alan C. Evans,et al.  Enhancement of MR Images Using Registration for Signal Averaging , 1998, Journal of Computer Assisted Tomography.

[85]  S. M. Aglioti,et al.  Mere observation of body discontinuity affects perceived ownership and vicarious agency over a virtual hand , 2015, Experimental Brain Research.

[86]  N. Yeung,et al.  Decision Processes in Human Performance Monitoring , 2010, The Journal of Neuroscience.

[87]  Joachim Lange,et al.  Beta oscillations define discrete perceptual cycles in the somatosensory domain , 2015, Proceedings of the National Academy of Sciences.

[88]  K. R. Ridderinkhof,et al.  Error-related brain potentials are differentially related to awareness of response errors: evidence from an antisaccade task. , 2001, Psychophysiology.

[89]  Stefan Koelsch,et al.  Processing Expectancy Violations during Music Performance and Perception: An ERP Study , 2010, Journal of Cognitive Neuroscience.

[90]  E Macaluso,et al.  Action anticipation beyond the action observation network: a functional magnetic resonance imaging study in expert basketball players , 2012, The European journal of neuroscience.

[91]  S. Rossi,et al.  Midfrontal theta transcranial alternating current stimulation modulates behavioural adjustment after error execution , 2018, The European journal of neuroscience.

[92]  Michael E J Masson,et al.  A tutorial on a practical Bayesian alternative to null-hypothesis significance testing , 2011, Behavior research methods.

[93]  H J Keselman,et al.  Repeated measures F tests and psychophysiological research: controlling the number of false positives. , 1980, Psychophysiology.

[94]  A. Delorme,et al.  Performance Monitoring Applied to System Supervision , 2017, Front. Hum. Neurosci..

[96]  G. Bird,et al.  Crossmodal Classification of Mu Rhythm Activity during Action Observation and Execution Suggests Specificity to Somatosensory Features of Actions , 2017, The Journal of Neuroscience.

[97]  Matteo Candidi,et al.  Theta synchronization over occipito‐temporal cortices during visual perception of body parts , 2018, The European journal of neuroscience.

[98]  Markus Ullsperger,et al.  Performance monitoring in the medial frontal cortex and related neural networks: From monitoring self actions to understanding others’ actions , 2018, Neuroscience Research.

[99]  Line Garnero,et al.  Inter-Brain Synchronization during Social Interaction , 2010, PloS one.

[100]  M. Lee,et al.  Bayesian Cognitive Modeling: A Practical Course , 2014 .

[101]  John J. B. Allen,et al.  Prelude to and Resolution of an Error: EEG Phase Synchrony Reveals Cognitive Control Dynamics during Action Monitoring , 2009, The Journal of Neuroscience.

[102]  Klaus Kessler,et al.  The Detection of Phase Amplitude Coupling during Sensory Processing , 2017, bioRxiv.

[103]  J. Hohnsbein,et al.  Effects of crossmodal divided attention on late ERP components. II. Error processing in choice reaction tasks. , 1991, Electroencephalography and clinical neurophysiology.

[104]  Stefan Everling,et al.  Theta-activity in anterior cingulate cortex predicts task rules and their adjustments following errors , 2010, Proceedings of the National Academy of Sciences.

[105]  M. Brass,et al.  When your error becomes my error: anterior insula activation in response to observed errors is modulated by agency. , 2016, Social cognitive and affective neuroscience.

[106]  S. Cochin,et al.  Observation and execution of movement: similarities demonstrated by quantified electroencephalography , 1999, The European journal of neuroscience.

[107]  Cosimo Urgesi,et al.  Vicarious motor activation during action perception: beyond correlational evidence , 2013, Front. Hum. Neurosci..

[108]  R. Hester,et al.  Error-related anterior cingulate cortex activity and the prediction of conscious error awareness , 2012, Front. Hum. Neurosci..

[109]  Jason S. Sherfey,et al.  Theta Oscillations Are Sensitive to Both Early and Late Conflict Processing Stages: Effects of Alcohol Intoxication , 2012, PloS one.

[110]  Harold Bekkering,et al.  Monitoring others' errors: The role of the motor system in early childhood and adulthood. , 2016, The British journal of developmental psychology.

[111]  J. Nadel,et al.  Revisiting mu suppression in autism spectrum disorder , 2014, Brain Research.

[112]  C. Urgesi,et al.  Simulating the future of actions in the human corticospinal system. , 2010, Cerebral cortex.

[113]  Moritz F. Wurm,et al.  Decoding Actions at Different Levels of Abstraction , 2015, The Journal of Neuroscience.

[114]  Jody C. Culham,et al.  fMRI Activation during Observation of Others' Reach Errors , 2010, Journal of Cognitive Neuroscience.

[115]  Robert Oostenveld,et al.  Brain symmetry and topographic analysis of lateralized event-related potentials , 2003, Clinical Neurophysiology.

[116]  J Hohnsbein,et al.  Effects of crossmodal divided attention on late ERP components. I. Simple and choice reaction tasks. , 1991, Electroencephalography and clinical neurophysiology.

[117]  S. Luck,et al.  How inappropriate high-pass filters can produce artifactual effects and incorrect conclusions in ERP studies of language and cognition. , 2015, Psychophysiology.

[118]  S. Swinnen,et al.  Neural correlates of action: Comparing meta-analyses of imagery, observation, and execution , 2018, Neuroscience & Biobehavioral Reviews.

[119]  Clay B. Holroyd,et al.  Hierarchical error processing: Different errors, different systems , 2007, Brain Research.

[120]  J. Decety,et al.  Oxytocin receptor gene variation predicts empathic concern and autonomic arousal while perceiving harm to others , 2014, Social neuroscience.

[121]  Marcel Brass,et al.  Observing accidental and intentional unusual actions is associated with different subregions of the medial frontal cortex , 2015, NeuroImage.

[122]  M. Benedek,et al.  EEG alpha synchronization is related to top-down processing in convergent and divergent thinking , 2011, Neuropsychologia.

[123]  Christian Keysers,et al.  Granger Causality Mapping during Joint Actions Reveals Evidence for Forward Models That Could Overcome Sensory-Motor Delays , 2010, PloS one.

[124]  G. Pezzulo,et al.  Interactional leader–follower sensorimotor communication strategies during repetitive joint actions , 2015, Journal of The Royal Society Interface.

[125]  Salvatore Maria Aglioti,et al.  Error, rather than its probability, elicits specific electrocortical signatures: a combined EEG-immersive virtual reality study of action observation. , 2018, Journal of neurophysiology.

[126]  G. Rizzolatti,et al.  Cortical mechanisms underlying the organization of goal-directed actions and mirror neuron-based action understanding. , 2014, Physiological reviews.

[127]  L. M. M.-T. Theory of Probability , 1929, Nature.

[128]  Michael Brady,et al.  Enhancement of MR Images , 1996, VBC.

[129]  H. Bekkering,et al.  Exploring the brain basis of joint action: Co-ordination of actions, goals and intentions , 2007, Social neuroscience.

[130]  Robert M. Kantner,et al.  An EEG Study , 1981 .

[131]  A. Nobre,et al.  Alpha Oscillations Related to Anticipatory Attention Follow Temporal Expectations , 2011, The Journal of Neuroscience.

[132]  Steen Moeller,et al.  The Human Connectome Project: A data acquisition perspective , 2012, NeuroImage.

[133]  D. Tucker,et al.  Frontal midline theta and the error-related negativity: neurophysiological mechanisms of action regulation , 2004, Clinical Neurophysiology.

[134]  Girijesh Prasad,et al.  Electrophysiological signatures of intentional social coordination in the 10–12Hz range , 2012, NeuroImage.

[135]  Lars Kai Hansen,et al.  Frontal alpha oscillations distinguish leaders from followers: Multivariate decoding of mutually interacting brains , 2014, NeuroImage.

[136]  M. Candidi,et al.  Visuo-motor interference with a virtual partner is equally present in cooperative and competitive interactions , 2018, Psychological Research.

[137]  Cosimo Urgesi,et al.  Neuroanatomical substrates of action perception and understanding: an anatomic likelihood estimation meta-analysis of lesion-symptom mapping studies in brain injured patients , 2014, Front. Hum. Neurosci..

[138]  Raymond H. Cuijpers,et al.  Joint Action: Neurocognitive Mechanisms Supporting Human Interaction , 2009, Top. Cogn. Sci..

[139]  K. R. Ridderinkhof,et al.  Not All Errors Are Alike: Theta and Alpha EEG Dynamics Relate to Differences in Error-Processing Dynamics , 2012, The Journal of Neuroscience.

[140]  Thomas Pinkpank,et al.  Does the error negativity reflect response conflict strength? Evidence from a Simon task. , 2007, Psychophysiology.

[141]  Mark C. W. van Rossum,et al.  Systematic biases in early ERP and ERF components as a result of high-pass filtering , 2012, Journal of Neuroscience Methods.

[142]  Ned Jenkinson,et al.  Dynamic Neural Correlates of Motor Error Monitoring and Adaptation during Trial-to-Trial Learning , 2014, The Journal of Neuroscience.

[143]  Blake W. Johnson,et al.  Mu rhythm modulation during observation of an object-directed grasp. , 2004, Brain research. Cognitive brain research.

[144]  Luca Turella,et al.  MEG Multivariate Analysis Reveals Early Abstract Action Representations in the Lateral Occipitotemporal Cortex , 2015, The Journal of Neuroscience.

[145]  Arnaud Delorme,et al.  Human or not human? Performance monitoring ERPs during human agent and machine supervision , 2019, NeuroImage.

[146]  Salvatore Maria Aglioti,et al.  The bright and the dark sides of motor simulation , 2017, Neuropsychologia.

[147]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.

[148]  Peter E. Keller,et al.  Neural alpha oscillations index the balance between self-other integration and segregation in real-time joint action , 2016, Neuropsychologia.

[149]  H. Bekkering,et al.  Joint action: bodies and minds moving together , 2006, Trends in Cognitive Sciences.