Decoding Complex Sounds Using Broadband Population Recordings from Secondary Auditory Cortex of Macaques

Direct electronic communication with sensory areas of the neocortex is a challenging ambition for brain-computer interfaces. Here, we report the first successful neural decoding of English words with high intelligibility from intracortical spike-based neural population activity recorded from the secondary auditory cortex of macaques. We acquired 96-channel full-broadband population recordings using intracortical microelectrode arrays in the rostral and caudal parabelt regions of the superior temporal gyrus (STG). We leveraged a new neural processing toolkit to investigate the choice of decoding algorithm, neural preprocessing, audio representation, channel count, and array location on neural decoding performance. The results illuminated a view of the auditory cortex as a spatially distributed network and a general purpose processor of complex sounds. The presented spike-based machine learning neural decoding approach may further be useful in informing future encoding strategies to deliver direct auditory percepts to the brain as specific patterns of microstimulation.

[1]  Harold W. Noonan.  An initial survey , 2019, Personal Identity.

[2]  J. Rauschecker,et al.  Processing of complex sounds in the macaque nonprimary auditory cortex. , 1995, Science.

[3]  Sydney S. Cash,et al.  Decoding word and category-specific spatiotemporal representations from MEG and EEG , 2011, NeuroImage.

[4]  Surya Ganguli,et al.  A theory of multineuronal dimensionality, dynamics and measurement , 2017, bioRxiv.

[5]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[6]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[7]  Nikos K Logothetis,et al.  Optimizing the imaging of the monkey auditory cortex: sparse vs. continuous fMRI. , 2009, Magnetic resonance imaging.

[8]  J. Donoghue,et al.  Failure mode analysis of silicon-based intracortical microelectrode arrays in non-human primates , 2013, Journal of neural engineering.

[9]  Jesper Jensen,et al.  An Algorithm for Predicting the Intelligibility of Speech Masked by Modulated Noise Maskers , 2016, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[10]  Spencer Kellis,et al.  Decoding stimulus identity from multi-unit activity and local field potentials along the ventral auditory stream in the awake primate: implications for cortical neural prostheses , 2013, Journal of neural engineering.

[11]  Arto V. Nurmikko,et al.  FPGA implementation of deep-learning recurrent neural networks with sub-millisecond real-time latency for BCI-decoding of large-scale neural sensors (104 nodes) , 2018, 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[12]  R. Quian Quiroga,et al.  Unsupervised Spike Detection and Sorting with Wavelets and Superparamagnetic Clustering , 2004, Neural Computation.

[13]  Florian Mormann,et al.  Reliable Analysis of Single-Unit Recordings from the Human Brain under Noisy Conditions: Tracking Neurons over Hours , 2016, PloS one.

[14]  W. Penfield.  SOME MECHANISMS OF CONSCIOUSNESS DISCOVERED DURING ELECTRICAL STIMULATION OF THE BRAIN. , 1958, Proceedings of the National Academy of Sciences of the United States of America.

[15]  Ming Yin,et al.  A 100-Channel Hermetically Sealed Implantable Device for Chronic Wireless Neurosensing Applications , 2013, IEEE Transactions on Biomedical Circuits and Systems.

[16]  Charles E Schroeder,et al.  Auditory Properties in the Parabelt Regions of the Superior Temporal Gyrus in the Awake Macaque Monkey: An Initial Survey , 2015, The Journal of Neuroscience.

[17]  J. Kaas,et al.  Subdivisions of auditory cortex and ipsilateral cortical connections of the parabelt auditory cortex in macaque monkeys , 1998, The Journal of comparative neurology.

[18]  Biao Tian,et al.  Processing of frequency-modulated sounds in the lateral auditory belt cortex of the rhesus monkey. , 2004, Journal of neurophysiology.

[19]  Jae S. Lim,et al.  Signal estimation from modified short-time Fourier transform , 1983, ICASSP.

[20]  J. Rauschecker,et al.  Functional Specialization in Rhesus Monkey Auditory Cortex , 2001, Science.

[21]  E. B. Newman,et al.  A Scale for the Measurement of the Psychological Magnitude Pitch , 1937 .

[22]  Yonggang Huang,et al.  A high-density, high-channel count, multiplexed μECoG array for auditory-cortex recordings. , 2014, Journal of neurophysiology.

[23]  Joji Tsunada,et al.  Auditory cortical activity drives feedback-dependent vocal control in marmosets , 2018, Nature Communications.

[24]  John D. Simeral,et al.  BCI decoder performance comparison of an LSTM recurrent neural network and a Kalman filter in retrospective simulation , 2018, 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER).

[25]  M. Mishkin,et al.  Functional Mapping of the Primate Auditory System , 2003, Science.

[26]  Andrew R. Dykstra,et al.  Widespread Brain Areas Engaged during a Classical Auditory Streaming Task Revealed by Intracranial EEG , 2011, Front. Hum. Neurosci..

[27]  Keith Johnson,et al.  Phonetic Feature Encoding in Human Superior Temporal Gyrus , 2014, Science.

[28]  D. Kipke,et al.  Cortical microstimulation in auditory cortex of rat elicits best-frequency dependent behaviors , 2005, Journal of neural engineering.

[29]  Yao Lu,et al.  Wireless Neurosensor for Full-Spectrum Electrophysiology Recordings during Free Behavior , 2014, Neuron.

[30]  Naohisa Miyakawa,et al.  Sound Frequency Representation in the Auditory Cortex of the Common Marmoset Visualized Using Optical Intrinsic Signal Imaging , 2018, eNeuro.

[31]  E. Chang,et al.  Categorical Speech Representation in Human Superior Temporal Gyrus , 2010, Nature Neuroscience.

[32]  Jon H. Kaas,et al.  'What' and 'where' processing in auditory cortex , 1999, Nature Neuroscience.

[33]  Konrad P. Körding,et al.  Machine Learning for Neural Decoding , 2017, eNeuro.

[34]  Robert T. Knight,et al.  Rapid tuning shifts in human auditory cortex enhance speech intelligibility , 2016, Nature Communications.

[35]  Stephen V. David,et al.  The Essential Complexity of Auditory Receptive Fields , 2015, PLoS Comput. Biol..

[36]  S. David,et al.  Integration over Multiple Timescales in Primary Auditory Cortex , 2013, The Journal of Neuroscience.

[37]  B. Averbeck,et al.  The primate cortical auditory system and neural representation of conspecific vocalizations. , 2009, Annual review of neuroscience.

[38]  J. Rauschecker,et al.  Maps and streams in the auditory cortex: nonhuman primates illuminate human speech processing , 2009, Nature Neuroscience.

[39]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[40]  Y. Yao,et al.  On Early Stopping in Gradient Descent Learning , 2007 .

[41]  Brian N. Pasley,et al.  Reconstructing Speech from Human Auditory Cortex , 2012, PLoS biology.

[42]  D. Bendor,et al.  The neuronal representation of pitch in primate auditory cortex , 2005, Nature.

[43]  Bahar Khalighinejad,et al.  Towards reconstructing intelligible speech from the human auditory cortex , 2019, Scientific Reports.

[44]  Michael J. Black,et al.  Decoding Complete Reach and Grasp Actions from Local Primary Motor Cortex Populations , 2010, The Journal of Neuroscience.

[45]  Harold W. Noonan.  AN INITIAL SURVEY , 2003 .