The Neural Representation of Sequences: From Transition Probabilities to Algebraic Patterns and Linguistic Trees

A sequence of images, sounds, or words can be stored at several levels of detail, from specific items and their timing to abstract structure. We propose a taxonomy of five distinct cerebral mechanisms for sequence coding: transitions and timing knowledge, chunking, ordinal knowledge, algebraic patterns, and nested tree structures. In each case, we review the available experimental paradigms and list the behavioral and neural signatures of the systems involved. Tree structures require a specific recursive neural code, as yet unidentified by electrophysiology, possibly unique to humans, and which may explain the singularity of human language and cognition.

[1]  G L Brotzman Thiamine deficiency. , 1992, The Journal of the American Board of Family Practice.

[2]  Geoffrey E. Hinton Tensor Product Variable Binding and the Representation of Symbolic Structures in Connectionist Systems , 1991 .

[3]  S. Dehaene,et al.  A Hierarchy of Responses to Auditory Regularities in the Macaque Brain , 2014, The Journal of Neuroscience.

[4]  Peter Hagoort,et al.  Topographical functional connectivity pattern in the perisylvian language networks. , 2010, Cerebral cortex.

[5]  Tony A. Plate,et al.  Holographic reduced representations , 1995, IEEE Trans. Neural Networks.

[6]  M. Huertas,et al.  Visually Cued Action Timing in the Primary Visual Cortex , 2015, Neuron.

[7]  S. Dehaene,et al.  A hierarchy of cortical responses to sequence violations in three-month-old infants , 2014, Cognition.

[8]  Andrea Moro "Dynamic Antisymmetry: movement as a symmetry-breaking phenomenon" , 1997 .

[9]  J. Tanji,et al.  Categorization of behavioural sequences in the prefrontal cortex , 2007, Nature.

[10]  C. Gallistel,et al.  Mice take calculated risks , 2012, Proceedings of the National Academy of Sciences.

[11]  John W. Harwell,et al.  Similar patterns of cortical expansion during human development and evolution , 2010, Proceedings of the National Academy of Sciences.

[12]  G. McCarthy,et al.  Perceiving patterns in random series: dynamic processing of sequence in prefrontal cortex , 2002, Nature Neuroscience.

[13]  A. Friederici The brain basis of language processing: from structure to function. , 2011, Physiological reviews.

[14]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[15]  J. Changeux,et al.  A Neuronal Model of Predictive Coding Accounting for the Mismatch Negativity , 2012, The Journal of Neuroscience.

[16]  M. Hauser,et al.  Grammatical pattern learning by human infants and cotton-top tamarin monkeys , 2008, Cognition.

[17]  Derek C. Penn,et al.  Darwin's mistake: Explaining the discontinuity between human and nonhuman minds , 2008, Behavioral and Brain Sciences.

[18]  Peter F MacNeilage Faculty Opinions recommendation of Functional neuroimaging of speech perception in infants. , 2002 .

[19]  S. Dehaene,et al.  Functional Neuroimaging of Speech Perception in Infants , 2002, Science.

[20]  P. Morosan,et al.  Broca's Region: Novel Organizational Principles and Multiple Receptor Mapping , 2010, PLoS biology.

[21]  F. Velde,et al.  Neural blackboard architectures of combinatorial structures in cognition , 2006 .

[22]  Peter Hagoort,et al.  Retrieval and unification of syntactic structure in sentence comprehension: an FMRI study using word-category ambiguity. , 2009, Cerebral cortex.

[23]  J. T. Massey,et al.  Mental rotation of the neuronal population vector. , 1989, Science.

[24]  Scott P. Johnson,et al.  Visual statistical learning in the newborn infant , 2011, Cognition.

[25]  Jim M. Monti,et al.  Neural repetition suppression reflects fulfilled perceptual expectations , 2008, Nature Neuroscience.

[26]  Kazuo Okanoya,et al.  Revisiting the syntactic abilities of non-human animals: natural vocalizations and artificial grammar learning , 2012, Philosophical Transactions of the Royal Society B: Biological Sciences.

[27]  Carl R Olson,et al.  Rank signals in four areas of macaque frontal cortex during selection of actions and objects in serial order. , 2010, Journal of neurophysiology.

[28]  Timothy Q. Gentner,et al.  Recursive syntactic pattern learning by songbirds , 2006, Nature.

[29]  Frank Restle,et al.  Theory of Serial Pattern Learning: Structural Trees. , 1970 .

[30]  Benjamin J. Kraus,et al.  Hippocampal “Time Cells”: Time versus Path Integration , 2013, Neuron.

[31]  J. Saffran,et al.  From Syllables to Syntax: Multilevel Statistical Learning by 12-Month-Old Infants , 2003 .

[32]  Kyle S. Smith,et al.  A Dual Operator View of Habitual Behavior Reflecting Cortical and Striatal Dynamics , 2013, Neuron.

[33]  M. Srinivasan,et al.  The concepts of ‘sameness’ and ‘difference’ in an insect , 2001, Nature.

[34]  S. Pinker,et al.  The faculty of language: what's special about it? , 2005, Cognition.

[35]  Giorgio Vallortigara,et al.  Origins of spatial, temporal and numerical cognition: Insights from comparative psychology , 2010, Trends in Cognitive Sciences.

[36]  Julia L. Evans,et al.  Can Infants Map Meaning to Newly Segmented Words? , 2007, Psychological science.

[37]  Kentaro Abe,et al.  Songbirds possess the spontaneous ability to discriminate syntactic rules , 2011, Nature Neuroscience.

[38]  Rushen Shi,et al.  Representations of abstract grammatical feature agreement in young children. , 2015, Journal of child language.

[39]  R. Malach,et al.  Syntactic structure building in the anterior temporal lobe during natural story listening , 2012, Brain and Language.

[40]  W. Fitch,et al.  Computational Constraints on Syntactic Processing in a Nonhuman Primate , 2004, Science.

[41]  G. Marcus The Algebraic Mind: Integrating Connectionism and Cognitive Science , 2001 .

[42]  Steven M Frankland,et al.  An architecture for encoding sentence meaning in left mid-superior temporal cortex , 2015, Proceedings of the National Academy of Sciences.

[43]  Xin Jin,et al.  Basal Ganglia Subcircuits Distinctively Encode the Parsing and Concatenation of Action Sequences , 2014, Nature Neuroscience.

[44]  Ghislaine Dehaene-Lambertz,et al.  Two-year-olds compute syntactic structure on-line. , 2010, Developmental science.

[45]  S. Dehaene,et al.  Event-Related Potential, Time-frequency, and Functional Connectivity Facets of Local and Global Auditory Novelty Processing: An Intracranial Study in Humans. , 2015, Cerebral cortex.

[46]  Andreas Nieder,et al.  Abstract rule neurons in the endbrain support intelligent behaviour in corvid songbirds , 2013, Nature Communications.

[47]  Peter Ford Dominey,et al.  Neural network processing of natural language: II. Towards a unified model of corticostriatal function in learning sentence comprehension and non-linguistic sequencing , 2009, Brain and Language.

[48]  Ann M Graybiel,et al.  Neural representation of time in cortico-basal ganglia circuits , 2009, Proceedings of the National Academy of Sciences.

[49]  D. Heeger,et al.  A Hierarchy of Temporal Receptive Windows in Human Cortex , 2008, The Journal of Neuroscience.

[50]  Karl J. Friston,et al.  A theory of cortical responses , 2005, Philosophical Transactions of the Royal Society B: Biological Sciences.

[51]  M. Shadlen,et al.  Representation of Time by Neurons in the Posterior Parietal Cortex of the Macaque , 2003, Neuron.

[52]  F. van der Velde,et al.  Neural blackboard architectures of combinatorial structures in cognition , 2006, Behavioral and Brain Sciences.

[53]  Hugo Merchant,et al.  Information Processing in the Primate Basal Ganglia during Sensory-Guided and Internally Driven Rhythmic Tapping , 2014, The Journal of Neuroscience.

[54]  Jim M. Monti,et al.  Expectation and Surprise Determine Neural Population Responses in the Ventral Visual Stream , 2010, The Journal of Neuroscience.

[55]  Daeyeol Lee,et al.  Order-Dependent Modulation of Directional Signals in the Supplementary and Presupplementary Motor Areas , 2007, The Journal of Neuroscience.

[56]  Marco Buiatti,et al.  Investigating the neural correlates of continuous speech computation with frequency-tagged neuroelectric responses , 2009, NeuroImage.

[57]  M. Sigman,et al.  Functional organization of perisylvian activation during presentation of sentences in preverbal infants , 2006, Proceedings of the National Academy of Sciences.

[58]  C. Olson,et al.  Statistical learning of visual transitions in monkey inferotemporal cortex , 2011, Proceedings of the National Academy of Sciences.

[59]  E. Spelke,et al.  Newborn infants perceive abstract numbers , 2009, Proceedings of the National Academy of Sciences.

[60]  A. Graybiel The Basal Ganglia and Chunking of Action Repertoires , 1998, Neurobiology of Learning and Memory.

[61]  Lorraine K. Tyler,et al.  Dynamic Processing in the Human Language System: Synergy between the Arcuate Fascicle and Extreme Capsule , 2011, The Journal of Neuroscience.

[62]  J. Duncan,et al.  Encoding Strategies Dissociate Prefrontal Activity from Working Memory Demand , 2003, Neuron.

[63]  K. Lashley The problem of serial order in behavior , 1951 .

[64]  Richard N. Aslin,et al.  The neural correlates of statistical learning in a word segmentation task: An fMRI study , 2013, Brain and Language.

[65]  Joël Fagot,et al.  The Temporal Dynamics of Regularity Extraction in Non-Human Primates , 2016, Cogn. Sci..

[66]  Charles Yang,et al.  Ontogeny and phylogeny of language , 2013, Proceedings of the National Academy of Sciences.

[67]  Robert F. Hadley The Problem of Rapid Variable Creation , 2009, Neural Computation.

[68]  Kazuo Okanoya,et al.  Birdsong neurolinguistics: songbird context-free grammar claim is premature , 2012, Neuroreport.

[69]  Denise Brandão de Oliveira e Britto,et al.  The faculty of language , 2007 .

[70]  D. Perani,et al.  Syntax and the Brain: Disentangling Grammar by Selective Anomalies , 2001, NeuroImage.

[71]  J. R. King,et al.  Single-trial decoding of auditory novelty responses facilitates the detection of residual consciousness , 2013, NeuroImage.

[72]  A. Mikami,et al.  Prefrontal activity during serial probe reproduction task: encoding, mnemonic, and retrieval processes. , 2006, Journal of neurophysiology.

[73]  Tristan A. Chaplin,et al.  A Conserved Pattern of Differential Expansion of Cortical Areas in Simian Primates , 2013, The Journal of Neuroscience.

[74]  Michael C. Frank,et al.  Three ideal observer models for rule learning in simple languages , 2011, Cognition.

[75]  H. Terrace,et al.  Knowledge of the Ordinal Position of List Items in Rhesus Monkeys , 1997 .

[76]  C. Gallistel,et al.  Toward a neurobiology of temporal cognition: advances and challenges , 1997, Current Opinion in Neurobiology.

[77]  A. Graybiel,et al.  Representation of Action Sequence Boundaries by Macaque Prefrontal Cortical Neurons , 2003, Science.

[78]  Jenny R. Saffran,et al.  Linking sounds to meanings: Infant statistical learning in a natural language , 2011, Cognitive Psychology.

[79]  A. Anwander,et al.  The brain differentiates human and non-human grammars: Functional localization and structural connectivity , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[80]  J. Mäkelä,et al.  Human auditory cortex is activated by omissions of auditory stimuli , 1997, Brain Research.

[81]  Angela D. Friederici,et al.  Hierarchical artificial grammar processing engages Broca's area , 2008, NeuroImage.

[82]  W Tecumseh Fitch,et al.  Toward a computational framework for cognitive biology: unifying approaches from cognitive neuroscience and comparative cognition. , 2014, Physics of life reviews.

[83]  F. Restle,et al.  Serial pattern learning: Pretraining of runs and trills , 1970 .

[84]  Elizabeth M. Brannon,et al.  Serial Expertise of Rhesus Macaques , 2003, Psychological science.

[85]  S. Dehaene,et al.  Cortical representation of the constituent structure of sentences , 2011, Proceedings of the National Academy of Sciences.

[86]  Dustin J. Merritt,et al.  Mechanisms of inferential order judgments in humans (Homo sapiens) and rhesus monkeys (Macaca mulatta). , 2011, Journal of comparative psychology.

[87]  Stefan Klöppel,et al.  Combining Functional and Anatomical Connectivity Reveals Brain Networks for Auditory Language Comprehension , 2022 .

[88]  Pooja Viswanathan,et al.  Neuronal correlates of a visual “sense of number” in primate parietal and prefrontal cortices , 2013, Proceedings of the National Academy of Sciences.

[89]  Angela D. Friederici,et al.  The language skeleton after dissecting meaning: A functional segregation within Broca's Area , 2015, NeuroImage.

[90]  R. Näätänen,et al.  The duration of a neuronal trace of an auditory stimulus as indicated by event-related potentials , 1987, Biological Psychology.

[91]  Yosef Grodzinsky,et al.  Neural correlates of syntactic movement: converging evidence from two fMRI experiments , 2004, NeuroImage.

[92]  S. Dehaene,et al.  Disruption of hierarchical predictive coding during sleep , 2015, Proceedings of the National Academy of Sciences.

[93]  János Horváth,et al.  How the human auditory system treats repetition amongst change , 2004, Neuroscience Letters.

[94]  Andreas Nieder,et al.  Supramodal numerosity selectivity of neurons in primate prefrontal and posterior parietal cortices , 2012, Proceedings of the National Academy of Sciences.

[95]  Hugo Merchant,et al.  Measuring time with different neural chronometers during a synchronization-continuation task , 2011, Proceedings of the National Academy of Sciences.

[96]  A. Todorović,et al.  Repetition Suppression and Expectation Suppression Are Dissociable in Time in Early Auditory Evoked Fields , 2012, The Journal of Neuroscience.

[97]  Naama Friedmann,et al.  An fMRI study of syntactic layers: Sentential and lexical aspects of embedding , 2009, NeuroImage.

[98]  Mark F. Bear,et al.  Learned spatiotemporal sequence recognition and prediction in primary visual cortex , 2014, Nature Neuroscience.

[99]  A. Syrota,et al.  The Cortical Representation of Speech , 1993, Journal of Cognitive Neuroscience.

[100]  Willem H. Zuidema,et al.  Simple rules can explain discrimination of putative recursive syntactic structures by a songbird species , 2009, Proceedings of the National Academy of Sciences.

[101]  Naama Friedmann,et al.  The crucial role of thiamine in the development of syntax and lexical retrieval: a study of infantile thiamine deficiency. , 2011, Brain : a journal of neurology.

[102]  S. Dehaene,et al.  Evidence for a hierarchy of predictions and prediction errors in human cortex , 2011, Proceedings of the National Academy of Sciences.

[103]  Daniel N. Osherson,et al.  Thought Beyond Language , 2012, Psychological science.

[104]  S. Koelsch Neural substrates of processing syntax and semantics in music , 2005, Current Opinion in Neurobiology.

[105]  Kuniyoshi L. Sakai,et al.  Neural Mechanisms Underlying the Computation of Hierarchical Tree Structures in Mathematics , 2014, PloS one.

[106]  Yukiko Kikuchi,et al.  Auditory Artificial Grammar Learning in Macaque and Marmoset Monkeys , 2013, The Journal of Neuroscience.

[107]  Shaul Hochstein,et al.  Macaque monkeys categorize images by their ordinal number , 2000, Nature.

[108]  Timothy E. J. Behrens,et al.  The evolution of the arcuate fasciculus revealed with comparative DTI , 2008, Nature Neuroscience.

[109]  Adam G. Thomas,et al.  Comparison of Human Ventral Frontal Cortex Areas for Cognitive Control and Language with Areas in Monkey Frontal Cortex , 2014, Neuron.

[110]  Scott P. Johnson,et al.  Infant Rule Learning Facilitated by Speech , 2007, Psychological science.

[111]  Naama Friedmann,et al.  The processing of different syntactic structures: fMRI investigation of the linguistic distinction between wh-movement and verb movement , 2014, Journal of Neurolinguistics.

[112]  P. Schoenemann,et al.  Prefrontal white matter volume is disproportionately larger in humans than in other primates , 2005, Nature Neuroscience.

[113]  P. Jusczyk,et al.  Infants' sensitivity to phonotactic patterns in the native language. , 1994 .

[114]  Trevor Bekolay,et al.  A Large-Scale Model of the Functioning Brain , 2012, Science.

[115]  E. Koechlin,et al.  Broca's Area and the Hierarchical Organization of Human Behavior , 2006, Neuron.

[116]  E. Spelke,et al.  Number-Space Mapping in Human Infants , 2010, Psychological science.

[117]  J. Gibbon Scalar expectancy theory and Weber's law in animal timing. , 1977 .

[118]  H. Tiitinen,et al.  Mismatch negativity (MMN), the deviance-elicited auditory deflection, explained. , 2010, Psychophysiology.

[119]  Stanislas Dehaene,et al.  Two Distinct Dynamic Modes Subtend the Detection of Unexpected Sounds , 2014, PloS one.

[120]  Anthony S. Kroch,et al.  The Linguistic Relevance of Tree Adjoining Grammar , 1985 .

[121]  Andreas Nieder,et al.  Temporal and Spatial Enumeration Processes in the Primate Parietal Cortex , 2006, Science.

[122]  永福 智志 The Organization of Learning , 2005, Journal of Cognitive Neuroscience.

[123]  Ramon Bartolo,et al.  Dynamic Representation of the Temporal and Sequential Structure of Rhythmic Movements in the Primate Medial Premotor Cortex , 2014, The Journal of Neuroscience.

[124]  Shogo Ohmae,et al.  Temporally Specific Sensory Signals for the Detection of Stimulus Omission in the Primate Deep Cerebellar Nuclei , 2013, The Journal of Neuroscience.

[125]  Joseph J. Paton,et al.  A Scalable Population Code for Time in the Striatum , 2015, Current Biology.

[126]  M. Corbetta,et al.  Evolutionarily Novel Functional Networks in the Human Brain? , 2013, The Journal of Neuroscience.

[127]  C. Büchel,et al.  Broca's area and the language instinct , 2003, Nature Neuroscience.

[128]  M. Giurfa,et al.  Simultaneous mastering of two abstract concepts by the miniature brain of bees , 2012, Proceedings of the National Academy of Sciences.

[129]  C. Schroeder,et al.  Role of cortical N-methyl-D-aspartate receptors in auditory sensory memory and mismatch negativity generation: implications for schizophrenia. , 1996, Proceedings of the National Academy of Sciences of the United States of America.

[130]  Angela R. Laird,et al.  Tackling the multifunctional nature of Broca's region meta-analytically: Co-activation-based parcellation of area 44 , 2013, NeuroImage.

[131]  William D. Marslen-Wilson,et al.  Left inferior frontal cortex and syntax: function, structure and behaviour in patients with left hemisphere damage , 2011, Brain : a journal of neurology.

[132]  Marco Zorzi,et al.  Spontaneous non-verbal counting in toddlers. , 2016, Developmental science.

[133]  J. Tanji,et al.  Integration of temporal order and object information in the monkey lateral prefrontal cortex. , 2004, Journal of neurophysiology.

[134]  Karl J. Friston,et al.  The mismatch negativity: A review of underlying mechanisms , 2009, Clinical Neurophysiology.

[135]  M. Botvinick,et al.  Neural representations of events arise from temporal community structure , 2013, Nature Neuroscience.

[136]  Angela D. Friederici,et al.  Artificial grammar learning meets formal language theory: an overview , 2012, Philosophical Transactions of the Royal Society B: Biological Sciences.

[137]  G. Hesslow,et al.  Memory trace and timing mechanism localized to cerebellar Purkinje cells , 2014, Proceedings of the National Academy of Sciences.

[138]  F. Perrin,et al.  Separate Representation of Stimulus Frequency, Intensity, and Duration in Auditory Sensory Memory: An Event-Related Potential and Dipole-Model Analysis , 1995, Journal of Cognitive Neuroscience.

[139]  Peter Hagoort,et al.  Syntactic Unification Operations Are Reflected in Oscillatory Dynamics during On-line Sentence Comprehension , 2010, Journal of Cognitive Neuroscience.

[140]  Noam Chomsky,et al.  Problems of projection , 2013 .

[141]  Alaa A. Kharbouch,et al.  Three models for the description of language , 1956, IRE Trans. Inf. Theory.

[142]  Stanislas Dehaene,et al.  Probing the lifetimes of auditory novelty detection processes , 2010, Neuropsychologia.

[143]  Marina Nespor,et al.  Perceptual and memory constraints on language acquisition , 2009, Trends in Cognitive Sciences.

[144]  J. Joseph,et al.  Prefrontal cortex and spatial sequencing in macaque monkey , 2004, Experimental Brain Research.

[145]  Matthew Botvinick,et al.  From Numerosity to Ordinal Rank: A Gain-Field Model of Serial Order Representation in Cortical Working Memory , 2007, The Journal of Neuroscience.

[146]  Stanislas Dehaene,et al.  A Temporal Bottleneck in the Language Comprehension Network , 2012, The Journal of Neuroscience.

[147]  S. Dehaene,et al.  Representation of Numerical and Sequential Patterns in Macaque and Human Brains , 2015, Current Biology.

[148]  Noam Chomsky,et al.  The faculty of language: what is it, who has it, and how did it evolve? , 2002, Science.

[149]  R N Aslin,et al.  Statistical Learning by 8-Month-Old Infants , 1996, Science.

[150]  C. Gallistel,et al.  Risk assessment in man and mouse , 2009, Proceedings of the National Academy of Sciences.

[151]  Peter M. Vishton,et al.  Rule learning by seven-month-old infants. , 1999, Science.

[152]  Mariano Sigman,et al.  The cortical representation of simple mathematical expressions , 2012, NeuroImage.

[153]  W. Newsome,et al.  The temporal precision of reward prediction in dopamine neurons , 2008, Nature Neuroscience.

[154]  C. Packer,et al.  Roaring and numerical assessment in contests between groups of female lions, Panthera leo , 1994, Animal Behaviour.

[155]  D. Osherson,et al.  The boundaries of language and thought in deductive inference , 2009, Proceedings of the National Academy of Sciences.

[156]  Dean V. Buonomano,et al.  ROBUST TIMING AND MOTOR PATTERNS BY TAMING CHAOS IN RECURRENT NEURAL NETWORKS , 2012, Nature Neuroscience.

[157]  Edward P. Stabler,et al.  An Introduction To Syntactic Analysis And Theory , 2016 .

[158]  Greg Jensen,et al.  Transfer of a Serial Representation between Two Distinct Tasks by Rhesus Macaques , 2013, PloS one.

[159]  W. Newsome,et al.  Context-dependent computation by recurrent dynamics in prefrontal cortex , 2013, Nature.

[160]  R. M. Siegel,et al.  Encoding of spatial location by posterior parietal neurons. , 1985, Science.

[161]  S. Dehaene,et al.  Neural signature of the conscious processing of auditory regularities , 2009, Proceedings of the National Academy of Sciences.

[162]  Jean-Francois Mangin,et al.  A robust cerebral asymmetry in the infant brain: The rightward superior temporal sulcus , 2011, NeuroImage.

[163]  Richard N Aslin,et al.  Bayesian learning of visual chunks by human observers , 2008, Proceedings of the National Academy of Sciences.

[164]  Marco Baroni,et al.  Processing of speech and non-speech sounds in the supratemporal plane: Auditory input preference does not predict sensitivity to statistical structure , 2013, NeuroImage.

[165]  Carl R Olson,et al.  Statistical Learning of Serial Visual Transitions by Neurons in Monkey Inferotemporal Cortex , 2014, The Journal of Neuroscience.

[166]  Hugo Merchant,et al.  Neurophysiology of timing in the hundreds of milliseconds: multiple layers of neuronal clocks in the medial premotor areas. , 2014, Advances in experimental medicine and biology.