Behavioral and Neural Variability of Naturalistic Arm Movements

Motor behaviors are central to many functions and dysfunctions of the brain, and understanding their neural basis has consequently been a major focus in neuroscience. However, most studies of motor behaviors have been restricted to artificial, repetitive paradigms, far removed from natural movements performed “in the wild.” Here, we leveraged recent advances in machine learning and computer vision to analyze intracranial recordings from 12 human subjects during thousands of spontaneous, unstructured arm reach movements, observed over several days for each subject. These naturalistic movements elicited cortical spectral power patterns consistent with findings from controlled paradigms, but with considerable neural variability across subjects and events. We modeled inter-event variability using ten behavioral and environmental features; the most important features explaining this variability were reach angle and day of recording. Our work is among the first studies connecting behavioral and neural variability across cortex in humans during unstructured movements and contributes to our understanding of long-term naturalistic behavior.

[1]  E. Keefer,et al.  Human motor decoding from neural signals: a review , 2019, BMC biomedical engineering.

[2]  Naotaka Fujii,et al.  Frequency-dependent spatiotemporal profiles of visual responses recorded with subdural ECoG electrodes in awake monkeys: Differences between high- and low-frequency activity , 2016, NeuroImage.

[3]  Christian K. Machens,et al.  Linearity of Cortical Receptive Fields Measured with Natural Sounds , 2004, The Journal of Neuroscience.

[4]  A. Georgopoulos,et al.  Mapping of the preferred direction in the motor cortex , 2007, Proceedings of the National Academy of Sciences.

[5]  F. Mosteller,et al.  Understanding robust and exploratory data analysis , 1985 .

[6]  E. Fetz Operant Conditioning of Cortical Unit Activity , 1969, Science.

[7]  Dawn M. Taylor,et al.  Direct Cortical Control of 3D Neuroprosthetic Devices , 2002, Science.

[8]  Martin Luessi,et al.  MEG and EEG data analysis with MNE-Python , 2013, Front. Neuroinform..

[9]  Mohsen Jamali,et al.  Decoding unconstrained arm movements in primates using high-density electrocorticography signals for brain-machine interface use , 2018, Scientific Reports.

[10]  Andreas Schulze-Bonhage,et al.  Signal quality of simultaneously recorded invasive and non-invasive EEG , 2009, NeuroImage.

[11]  Ilya Nemenman,et al.  Millisecond Spike Timing Codes for Motor Control , 2018, Trends in Neurosciences.

[12]  Kai J Miller,et al.  A library of human electrocorticographic data and analyses , 2019, Nature Human Behaviour.

[13]  Kathryn Bonnen,et al.  Beyond Trial-Based Paradigms: Continuous Behavior, Ongoing Neural Activity, and Natural Stimuli , 2018, The Journal of Neuroscience.

[14]  David I. Donaldson,et al.  Understanding Minds in Real-World Environments: Toward a Mobile Cognition Approach , 2017, Front. Hum. Neurosci..

[15]  Robert T. Knight,et al.  Encoding and Decoding Models in Cognitive Electrophysiology , 2017, Front. Syst. Neurosci..

[16]  Emilio Bizzi,et al.  Discharge of frontal eye field neurons during saccadic and following eye movements in unanesthetized monkeys , 1968, Experimental Brain Research.

[17]  Matthew T. Kaufman,et al.  Single-trial neural dynamics are dominated by richly varied movements , 2019, Nature Neuroscience.

[18]  Jack L. Gallant,et al.  Encoding and decoding in fMRI , 2011, NeuroImage.

[19]  W. David Hairston,et al.  Systems, Subjects, Sessions: To What Extent Do These Factors Influence EEG Data? , 2017, Front. Hum. Neurosci..

[20]  E. Fetz,et al.  Correlations between the same motor cortex cells and arm muscles during a trained task, free behavior, and natural sleep in the macaque monkey. , 2007, Journal of neurophysiology.

[21]  S. Cash,et al.  The number of seizures needed in the EMU , 2015, Epilepsia.

[22]  Stephen G Lisberger,et al.  How and why neural and motor variation are related , 2015, Current Opinion in Neurobiology.

[23]  E. Basar,et al.  Gamma, alpha, delta, and theta oscillations govern cognitive processes. , 2001, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[24]  Charles H. Brown,et al.  The Influence of Natural Scene Dynamics on Auditory Cortical Activity , 2010, The Journal of Neuroscience.

[25]  Daniel P. Ferris,et al.  Electrocortical activity is coupled to gait cycle phase during treadmill walking , 2011, NeuroImage.

[26]  Doris Y. Tsao,et al.  The Code for Facial Identity in the Primate Brain , 2017, Cell.

[27]  Daniel P Ferris,et al.  Differentiation in Theta and Beta Electrocortical Activity between Visual and Physical Perturbations to Walking and Standing Balance , 2018, eNeuro.

[28]  Reza Shadmehr,et al.  Motor variability is not noise, but grist for the learning mill , 2014, Nature Neuroscience.

[29]  W. David Hairston,et al.  Human electrocortical dynamics while stepping over obstacles , 2019, Scientific Reports.

[30]  Michael Stuart,et al.  Understanding Robust and Exploratory Data Analysis , 1984 .

[31]  Yuxiao Yang,et al.  Dynamic tracking of non-stationarity in human ECoG activity , 2017, 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[32]  Mohammad Dastjerdi,et al.  Numerical processing in the human parietal cortex during experimental and natural conditions , 2013, Nature Communications.

[33]  Uri Hasson,et al.  Keep it real: rethinking the primacy of experimental control in cognitive neuroscience , 2020, NeuroImage.

[34]  Shachar Maidenbaum,et al.  Grid-like hexadirectional modulation of human entorhinal theta oscillations , 2018, Proceedings of the National Academy of Sciences.

[35]  Sidrat Tasawoor Kanth,et al.  Electrocorticogram (ECoG) Is Highly Informative in Primate Visual Cortex , 2020, The Journal of Neuroscience.

[36]  Jason Omedes,et al.  Factors that affect error potentials during a grasping task: toward a hybrid natural movement decoding BCI , 2018, Journal of neural engineering.

[37]  E. Vaadia,et al.  Single-unit activity related to bimanual arm movements in the primary and supplementary motor cortices. , 2002, Journal of neurophysiology.

[38]  A. Engel,et al.  Beta-band oscillations—signalling the status quo? , 2010, Current Opinion in Neurobiology.

[39]  J. Gibbs Fourier's Series , 1899, Nature.

[40]  J. Donoghue,et al.  Primary Motor Cortex Tuning to Intended Movement Kinematics in Humans with Tetraplegia , 2008, The Journal of Neuroscience.

[41]  Yasuharu Koike,et al.  Prediction of Three-Dimensional Arm Trajectories Based on ECoG Signals Recorded from Human Sensorimotor Cortex , 2013, PloS one.

[42]  Yoshua Bengio,et al.  Adversarial Domain Adaptation for Stable Brain-Machine Interfaces , 2018, ICLR.

[43]  J. Assad,et al.  Putaminal activity for simple reactions or self-timed movements. , 2003, Journal of neurophysiology.

[44]  Rajesh P. N. Rao,et al.  Towards naturalistic human neuroscience and neuroengineering: behavior mining in long-term video and neural recordings , 2020, ArXiv.

[45]  Frederick R. Forst,et al.  On robust estimation of the location parameter , 1980 .

[46]  M. Orger,et al.  Whole-Brain Activity Maps Reveal Stereotyped, Distributed Networks for Visuomotor Behavior , 2014, Neuron.

[47]  Biyu J. He,et al.  Volition and Action in the Human Brain: Processes, Pathologies, and Reasons , 2017, The Journal of Neuroscience.

[48]  Krishna V. Shenoy,et al.  Challenges and Opportunities for Next-Generation Intracortically Based Neural Prostheses , 2011, IEEE Transactions on Biomedical Engineering.

[49]  Matthias Bethge,et al.  DeepLabCut: markerless pose estimation of user-defined body parts with deep learning , 2018, Nature Neuroscience.

[50]  H. Niki,et al.  Prefrontal cortical unit activity and delayed alternation performance in monkeys. , 1971, Journal of neurophysiology.

[51]  Jeremy R. Manning,et al.  Broadband Shifts in Local Field Potential Power Spectra Are Correlated with Single-Neuron Spiking in Humans , 2009, The Journal of Neuroscience.

[52]  N. Tzourio-Mazoyer,et al.  Automated Anatomical Labeling of Activations in SPM Using a Macroscopic Anatomical Parcellation of the MNI MRI Single-Subject Brain , 2002, NeuroImage.

[53]  Cory S. Inman,et al.  Single-Neuron Representations of Spatial Targets in Humans , 2019, Current Biology.

[54]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.

[55]  Gordon J. Berman,et al.  Measuring behavior across scales , 2017, BMC Biology.

[56]  Mariska J Vansteensel,et al.  High-frequency band temporal dynamics in response to a grasp force task , 2019, Journal of neural engineering.

[57]  E. Evarts A technique for recording activity of subcortical neurons in moving animals. , 1968, Electroencephalography and clinical neurophysiology.

[58]  Sonja Grün,et al.  Local field potentials in primate motor cortex encode grasp kinetic parameters , 2015, NeuroImage.

[59]  James C. Christensen,et al.  Day-to-day variability in hybrid, passive brain-computer interfaces: Comparing two studies assessing cognitive workload , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[60]  Kazuhiko Seki,et al.  Decoding of muscle activity from the sensorimotor cortex in freely behaving monkeys , 2019, NeuroImage.

[61]  D.A. Heldman,et al.  Local field potential spectral tuning in motor cortex during reaching , 2006, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[62]  D. Poeppel,et al.  Coupled neural systems underlie the production and comprehension of naturalistic narrative speech , 2014, Proceedings of the National Academy of Sciences.

[63]  Laurent Bougrain,et al.  Decoding Finger Flexion from Band-Specific ECoG Signals in Humans , 2012, Front. Neurosci..

[64]  Sabine Van Huffel,et al.  Mobile EEG on the bike: disentangling attentional and physical contributions to auditory attention tasks , 2016, Journal of neural engineering.

[65]  Robert Oostenveld,et al.  FieldTrip: Open Source Software for Advanced Analysis of MEG, EEG, and Invasive Electrophysiological Data , 2010, Comput. Intell. Neurosci..

[66]  A. Michelson,et al.  Fourier's Series , 1898, Nature.

[67]  Rajesh P. N. Rao,et al.  Cortical Topography of Error-Related High-Frequency Potentials During Erroneous Control in a Continuous Control Brain–Computer Interface , 2019, Front. Neurosci..

[68]  Opher Donchin,et al.  Individual Movement Variability Magnitudes Are Explained by Cortical Neural Variability , 2017, The Journal of Neuroscience.

[69]  Suely Master,et al.  O espectro médio de longo termo na pesquisa e na clínica fonoaudiológica , 2006 .

[70]  Marcel A. J. van Gerven,et al.  A primer on encoding models in sensory neuroscience , 2017 .

[71]  Lena H Ting,et al.  Neuromechanics of muscle synergies for posture and movement , 2007, Current Opinion in Neurobiology.

[72]  Lee E. Miller,et al.  Long-term stability of cortical population dynamics underlying consistent behavior , 2019, Nature Neuroscience.

[73]  Rajesh P. N. Rao,et al.  Unsupervised Decoding of Long-Term, Naturalistic Human Neural Recordings with Automated Video and Audio Annotations , 2015, Front. Hum. Neurosci..

[74]  Joanna E. M. Scanlon,et al.  Taking off the training wheels: Measuring auditory P3 during outdoor cycling using an active wet EEG system , 2017, Brain Research.

[75]  L. Snyder,et al.  Single Units in the Posterior Parietal Cortex Encode Patterns of Bimanual Coordination , 2018, Cerebral cortex.

[76]  Naotaka Fujii,et al.  Long-Term Asynchronous Decoding of Arm Motion Using Electrocorticographic Signals in Monkeys , 2009, Front. Neuroeng..

[77]  Clement Hamani,et al.  Temporal alignment of electrocorticographic recordings for upper limb movement , 2015, Front. Neurosci..

[78]  Caroline Szymanski,et al.  Teams on the same wavelength perform better: Inter-brain phase synchronization constitutes a neural substrate for social facilitation , 2017, NeuroImage.

[79]  P. J. Huber Robust Estimation of a Location Parameter , 1964 .

[80]  A. P. Georgopoulos,et al.  Neuronal population coding of movement direction. , 1986, Science.

[81]  Michael J. Kahana,et al.  Direct brain recordings fuel advances in cognitive electrophysiology , 2010, Trends in Cognitive Sciences.

[82]  D. Vaillancourt,et al.  Beta-band oscillations in the supplementary motor cortex are modulated by levodopa and associated with functional activity in the basal ganglia , 2018, NeuroImage: Clinical.

[83]  Mariska J Vansteensel,et al.  Encoding of kinetic and kinematic movement parameters in the sensorimotor cortex: A Brain‐Computer Interface perspective , 2019, The European journal of neuroscience.

[84]  Daniel P Ferris,et al.  Transient visual perturbations boost short-term balance learning in virtual reality by modulating electrocortical activity. , 2018, Journal of neurophysiology.

[85]  Bingni W. Brunton,et al.  Data-driven models in human neuroscience and neuroengineering , 2019, Current Opinion in Neurobiology.

[86]  Joseph J. Paton,et al.  Big behavioral data: psychology, ethology and the foundations of neuroscience , 2014, Nature Neuroscience.

[87]  Line Garnero,et al.  Inter-Brain Synchronization during Social Interaction , 2010, PloS one.

[88]  Nicholas Stergiou,et al.  Movement Variability and the Use of Nonlinear Tools: Principles to Guide Physical Therapist Practice , 2009, Physical Therapy.

[89]  P. Roelfsema,et al.  Alpha and gamma oscillations characterize feedback and feedforward processing in monkey visual cortex , 2014, Proceedings of the National Academy of Sciences.

[90]  Lucia Melloni,et al.  Neural correlates of unstructured motor behaviors , 2019, Journal of neural engineering.

[91]  Daniel A. Braun,et al.  Motor Task Variation Induces Structural Learning , 2009, Current Biology.

[92]  Benjamin L. de Bivort,et al.  Ethology as a physical science , 2018, Nature Physics.

[93]  Andreas Schulze-Bonhage,et al.  Decoding natural grasp types from human ECoG , 2012, NeuroImage.

[94]  Xi Jiang,et al.  Coarse behavioral context decoding , 2019, Journal of neural engineering.

[95]  J. Gallant,et al.  Natural Stimulus Statistics Alter the Receptive Field Structure of V1 Neurons , 2004, The Journal of Neuroscience.

[96]  J. A. Wilson,et al.  Two-dimensional movement control using electrocorticographic signals in humans , 2008, Journal of neural engineering.

[97]  J. Kalaska From intention to action: motor cortex and the control of reaching movements. , 2009, Advances in experimental medicine and biology.

[98]  E. Redcay,et al.  Perceived live interaction modulates the developing social brain. , 2016, Social cognitive and affective neuroscience.

[99]  Andreas Schulze-Bonhage,et al.  “Doctor” or “darling”? Decoding the communication partner from ECoG of the anterior temporal lobe during non-experimental, real-life social interaction , 2012, Front. Hum. Neurosci..

[100]  N. F. Ramsey,et al.  Task-free electrocorticography frequency mapping of the motor cortex , 2013, Clinical Neurophysiology.

[101]  David B. Grayden,et al.  Consistency of Long-Term Subdural Electrocorticography in Humans , 2017, IEEE Transactions on Biomedical Engineering.

[102]  E. Fetz,et al.  Decoupling the Cortical Power Spectrum Reveals Real-Time Representation of Individual Finger Movements in Humans , 2009, The Journal of Neuroscience.

[103]  N. D. De Biase,et al.  [The long-term average spectrum in research and in the clinical practice of speech therapists]. , 2006, Pro-fono : revista de atualizacao cientifica.

[104]  Qiang Ji,et al.  Decoding onset and direction of movements using Electrocorticographic (ECoG) signals in humans , 2012, Front. Neuroeng..

[105]  Oren Sagher,et al.  Variability in the location of high frequency oscillations during prolonged intracranial EEG recordings , 2018, Nature Communications.

[106]  Thomas L. Griffiths,et al.  Supplementary Information for Natural Speech Reveals the Semantic Maps That Tile Human Cerebral Cortex , 2022 .

[107]  Morgan D. Barense,et al.  Temporal integration of narrative information in a hippocampal amnesic patient , 2019, NeuroImage.

[108]  T. Tcheng,et al.  Clinical and electrocorticographic response to antiepileptic drugs in patients treated with responsive stimulation , 2018, Epilepsy & Behavior.

[109]  David J. Anderson,et al.  Computational Neuroethology: A Call to Action , 2019, Neuron.

[110]  P. Brown,et al.  Decoding gripping force based on local field potentials recorded from subthalamic nucleus in humans , 2016, eLife.

[111]  Jing Wu,et al.  Task-Specific Somatosensory Feedback via Cortical Stimulation in Humans , 2016, IEEE Transactions on Haptics.

[112]  Lüder Deecke,et al.  Brain potential changes in voluntary and passive movements in humans: readiness potential and reafferent potentials , 2016, Pflügers Archiv - European Journal of Physiology.

[113]  G. Schalk,et al.  Brain-Computer Interfaces Using Electrocorticographic Signals , 2011, IEEE Reviews in Biomedical Engineering.

[114]  Bijan Pesaran,et al.  Neural Correlates of Visual–Spatial Attention in Electrocorticographic Signals in Humans , 2011, Front. Hum. Neurosci..

[115]  Edward F. Chang,et al.  Speech synthesis from neural decoding of spoken sentences , 2019, Nature.

[116]  Kenneth Kreutz-Delgado,et al.  Measure projection analysis: A probabilistic approach to EEG source comparison and multi-subject inference , 2013, NeuroImage.

[117]  Ali Farhadi,et al.  AJILE Movement Prediction: Multimodal Deep Learning for Natural Human Neural Recordings and Video , 2017, AAAI.

[118]  Robert Oostenveld,et al.  Integrated analysis of anatomical and electrophysiological human intracranial data , 2017, Nature Protocols.

[119]  Stephen A. Coombes,et al.  3D Cortical electrophysiology of ballistic upper limb movement in humans , 2015, NeuroImage.

[120]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[121]  Gerwin Schalk,et al.  A brain–computer interface using electrocorticographic signals in humans , 2004, Journal of neural engineering.

[122]  David J. Anderson,et al.  Toward a Science of Computational Ethology , 2014, Neuron.

[123]  J. Gibbs Fourier's Series , 1898, Nature.

[124]  F. Paulus,et al.  The Social Neuroscience of Interpersonal Emotions. , 2016, Current topics in behavioral neurosciences.

[125]  W. Schultz,et al.  Neuronal activity preceding self-initiated or externally timed arm movements in area 6 of monkey cortex , 2004, Experimental Brain Research.

[126]  Andreas Schulze-Bonhage,et al.  Prediction of arm movement trajectories from ECoG-recordings in humans , 2008, Journal of Neuroscience Methods.

[127]  E. Redcay,et al.  Using second-person neuroscience to elucidate the mechanisms of social interaction , 2019, Nature Reviews Neuroscience.

[128]  Rajesh P. N. Rao,et al.  Spectral Changes in Cortical Surface Potentials during Motor Movement , 2007, The Journal of Neuroscience.

[129]  Asif A. Ghazanfar,et al.  The Natural Statistics of Audiovisual Speech , 2009, PLoS Comput. Biol..

[130]  E. Evarts,et al.  Relation of pyramidal tract activity to force exerted during voluntary movement. , 1968, Journal of neurophysiology.

[131]  Timothy J Ebner,et al.  Past, present, and emerging principles in the neural encoding of movement. , 2009, Advances in experimental medicine and biology.

[132]  Clement Hamani,et al.  Reconstruction of reaching movement trajectories using electrocorticographic signals in humans , 2017, PloS one.