Formal models of Structure Building in Music, Language and Animal Songs

Human language, music and a variety of animal vocalisations constitute ways of sonic communication that exhibit remarkable structural complexity. While the complexities of language and possible parallels in animal communication have been discussed intensively, reflections on the complexity of music and animal song, and their comparisons are underrepresented. In some ways, music and animal songs are more comparable to each other than to language, as propositional semantics cannot be used as as indicator of communicative success or well-formedness, and notions of grammaticality are less easily defined. This review brings together accounts of the principles of structure building in language, music and animal song, relating them to the corresponding models in formal language theory, with a special focus on evaluating the benefits of using the Chomsky hierarchy (CH). We further discuss common misunderstandings and shortcomings concerning the CH, as well as extensions or augmentations of it that address some of these issues, and suggest ways to move beyond.

[1]  Jun Wu Statistical language model , 2018, The Beauty of Mathematics in Computer Science.

[2]  Charles Ames,et al.  The Markov Process as a Compositional Model: A Survey and Tutorial , 2017 .

[3]  C. ten Cate Assessing the uniqueness of language: Animal grammatical abilities take center stage , 2016, Psychonomic bulletin & review.

[4]  Yan Huang,et al.  Bias and Agreement in Syntactic Annotations , 2016, ArXiv.

[5]  C. Scharff,et al.  Waltzing Taeniopygia: integration of courtship song and dance in the domesticated Australian zebra finch , 2016, Animal Behaviour.

[6]  V. Ivanitskii,et al.  Syntax of complex bird song in the large-billed reed warbler (Acrocephalus orinus) , 2016 .

[7]  L. Robert Slevc,et al.  Prosodic Structure as a Parallel to Musical Structure , 2015, Front. Psychol..

[8]  Carel ten Cate,et al.  Zebra finches are able to learn affixation-like patterns , 2015, Animal Cognition.

[9]  Alexander Clark,et al.  Unsupervised Prediction of Acceptability Judgements , 2015, ACL.

[10]  Fei-Fei Li,et al.  Visualizing and Understanding Recurrent Networks , 2015, ArXiv.

[11]  David J. Heeger,et al.  The neural processing of hierarchical structure in music and speech at different timescales , 2015, Front. Neurosci..

[12]  Geraint A. Wiggins,et al.  The evolutionary roots of creativity: mechanisms and motivations , 2014 .

[13]  Yoshua Bengio,et al.  Gated Feedback Recurrent Neural Networks , 2015, ICML.

[14]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[15]  Arik Kershenbaum,et al.  Animal vocal sequences: not the Markov chains we thought they were , 2014, Proceedings of the Royal Society B: Biological Sciences.

[16]  Carel ten Cate,et al.  On the phonetic and syntactic processing abilities of birds: From songs to speech and artificial grammars , 2014, Current Opinion in Neurobiology.

[17]  S. Koelsch,et al.  Tension-related activity in the orbitofrontal cortex and amygdala: an fMRI study with music. , 2014, Social cognitive and affective neuroscience.

[18]  Phong Le,et al.  The Inside-Outside Recursive Neural Network model for Dependency Parsing , 2014, EMNLP.

[19]  Jordan L. Boyd-Graber,et al.  A Neural Network for Factoid Question Answering over Paragraphs , 2014, EMNLP.

[20]  Mark Steedman,et al.  A Robust Parser-Interpreter for Jazz Chord Sequences , 2014 .

[21]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[22]  Carel ten Cate,et al.  Zebra finches are sensitive to prosodic features of human speech , 2014, Proceedings of the Royal Society B: Biological Sciences.

[23]  Constance Scharff,et al.  The use of network analysis to study complex animal communication systems: a study on nightingale song , 2014, Proceedings of the Royal Society B: Biological Sciences.

[24]  Quoc V. Le,et al.  Distributed Representations of Sentences and Documents , 2014, ICML.

[25]  Correction: Is "Huh?" a Universal Word? Conversational Infrastructure and the Convergent Evolution of Linguistic Items , 2014, PLoS ONE.

[26]  David Rothenberg,et al.  Investigation of musicality in birdsong , 2014, Hearing Research.

[27]  Kazuo Okanoya,et al.  A simple explanation for the evolution of complex song syntax in Bengalese finches , 2013, Biology Letters.

[28]  Yukiko Kikuchi,et al.  Auditory Artificial Grammar Learning in Macaque and Marmoset Monkeys , 2013, The Journal of Neuroscience.

[29]  Elizabeth Hellmuth Margulis,et al.  On Repeat: How Music Plays the Mind , 2013 .

[30]  Francisco Torreira,et al.  Is “Huh?” a Universal Word? Conversational Infrastructure and the Convergent Evolution of Linguistic Items , 2013, PloS one.

[31]  Chang Dong,et al.  Data Warehouse Tuning: The Supremacy of Bitmap Index , 2013 .

[32]  Benoît Fabre,et al.  New acoustic model for humpback whale sound production , 2013 .

[33]  Stefan Koelsch,et al.  Processing of hierarchical syntactic structure in music , 2013, Proceedings of the National Academy of Sciences.

[34]  Zoltan Dienes,et al.  The nature of the memory buffer in implicit learning: Learning Chinese tonal symmetries , 2013, Consciousness and Cognition.

[35]  Geraint A. Wiggins,et al.  Multiple Viewpoint Systems: Time Complexity and the Construction of Domains for Complex Musical Viewpoints in the Harmonization Problem , 2013 .

[36]  Andrew Y. Ng,et al.  Parsing with Compositional Vector Grammars , 2013, ACL.

[37]  D. Margoliash,et al.  A Mechanism for Frequency Modulation in Songbirds Shared with Humans , 2013, The Journal of Neuroscience.

[38]  Gary F. Marcus,et al.  Stepwise acquisition of vocal combinatorial capacity in songbirds and human infants , 2013, Nature.

[39]  Timothy J. Gardner,et al.  Long-range Order in Canary Song , 2013, PLoS Comput. Biol..

[40]  Matthew D. Zeiler ADADELTA: An Adaptive Learning Rate Method , 2012, ArXiv.

[41]  Martin Rohrmeier,et al.  Implicit Learning of Recursive Context-Free Grammars , 2012, PloS one.

[42]  M. Rohrmeier,et al.  Implicit Learning and Acquisition of Music , 2012, Top. Cogn. Sci..

[43]  Geraint A. Wiggins,et al.  Auditory Expectation: The Information Dynamics of Music Perception and Cognition , 2012, Top. Cogn. Sci..

[44]  Charles E. Taylor,et al.  Structural Design Principles of Complex Bird Songs: A Network-Based Approach , 2012, PloS one.

[45]  Peter Hagoort,et al.  Implicit Acquisition of Grammars With Crossed and Nested Non-Adjacent Dependencies: Investigating the Push-Down Stack Model , 2012, Cogn. Sci..

[46]  Kazuo Okanoya,et al.  Revisiting the syntactic abilities of non-human animals: natural vocalizations and artificial grammar learning , 2012, Philosophical Transactions of the Royal Society B: Biological Sciences.

[47]  James Rogers,et al.  Formal language theory: refining the Chomsky hierarchy , 2012, Philosophical Transactions of the Royal Society B: Biological Sciences.

[48]  Angela D. Friederici,et al.  Artificial grammar learning meets formal language theory: an overview , 2012, Philosophical Transactions of the Royal Society B: Biological Sciences.

[49]  Peter Hagoort,et al.  Pattern perception and computational complexity: introduction to the special issue , 2012, Philosophical Transactions of the Royal Society B: Biological Sciences.

[50]  Lei Zhu,et al.  Unconscious structural knowledge of tonal symmetry: Tang poetry redefines limits of implicit learning , 2012, Consciousness and Cognition.

[51]  K. Zuberbühler,et al.  Call combinations in monkeys: Compositional or idiomatic expressions? , 2012, Brain and Language.

[52]  S. Koelsch,et al.  Predictive information processing in music cognition. A critical review. , 2012, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[53]  Amos J. Storkey,et al.  Comparing Probabilistic Models for Melodic Sequences , 2011, ECML/PKDD.

[54]  Frank A. Russo,et al.  The motor origins of human and avian song structure , 2011, Proceedings of the National Academy of Sciences.

[55]  Kentaro Abe,et al.  Songbirds possess the spontaneous ability to discriminate syntactic rules , 2011, Nature Neuroscience.

[56]  Emmanuel Vincent,et al.  Dynamic Bayesian Networks for Symbolic Polyphonic Pitch Modeling , 2011, IEEE Transactions on Audio, Speech, and Language Processing.

[57]  C. Scharff,et al.  Evo-devo, deep homology and FoxP2: implications for the evolution of speech and language , 2011, Philosophical Transactions of the Royal Society B: Biological Sciences.

[58]  Panayotis Mavromatis,et al.  Voice-Leading Prototypes and Harmonic Function in Two Chorale Corpora , 2011, MCM.

[59]  Martin Rohrmeier,et al.  Exploring Rameau and Beyond: A Corpus Study of Root Progression Theories , 2011, MCM.

[60]  Mitsuko Aramaki,et al.  Is the distinction between intra- and extra-musical meaning implemented in the brain? Comment on "Towards a neural basis of processing musical semantics" by Stefan Koelsch. , 2011, Physics of life reviews.

[61]  S. Koelsch Towards a neural basis of processing musical semantics. , 2011, Physics of life reviews.

[62]  W. Fitch,et al.  Multiple varieties of musical meaning: Comment on "Towards a neural basis of processing musical semantics" by Stefan Koelsch. , 2011, Physics of life reviews.

[63]  S. Davies Questioning the distinction between intra- and extra-musical meaning: Comment on "Towards a neural basis for processing musical semantics" by Stefan Koelsch. , 2011, Physics of life reviews.

[64]  Uli Reich,et al.  The meanings of semantics: Comment on "Towards a neural basis of processing musical semantics" by Stefan Koelsch. , 2011, Physics of life reviews.

[65]  L. Robert Slevc,et al.  Meaning in music and language: Three key differences: Comment on "Towards a neural basis of processing musical semantics" by Stefan Koelsch. , 2011, Physics of life reviews.

[66]  Martin Rohrmeier,et al.  Towards a generative syntax of tonal harmony , 2011 .

[67]  R. Berwick,et al.  Songs to syntax: the linguistics of birdsong , 2011, Trends in Cognitive Sciences.

[68]  Trevor de Clercq,et al.  A corpus analysis of rock harmony , 2011, Popular Music.

[69]  Dezhe Z. Jin,et al.  A Compact Statistical Model of the Song Syntax in Bengalese Finch , 2010, PLoS Comput. Biol..

[70]  Masato Okada,et al.  Complex Sequencing Rules of Birdsong Can be Explained by Simple Hidden Markov Processes , 2010, PloS one.

[71]  C. Scharff,et al.  Twitter evolution: converging mechanisms in birdsong and human speech , 2010, Nature Reviews Neuroscience.

[72]  Peter Marler,et al.  The Organization of Song Repertoires in Song Sparrows: Themes and Variations , 2010 .

[73]  Roger Payne,et al.  Large Scale Changes over 19 Years in Songs of Humpback Whales in Bermuda , 2010 .

[74]  Richard L. Lewis,et al.  Short-term forgetting in sentence comprehension: Crosslinguistic evidence from verb-final structures , 2010 .

[75]  Thierry Aubin,et al.  Are bird song complexity and song sharing shaped by habitat structure? An information theory and statistical approach. , 2010, Journal of theoretical biology.

[76]  Willem H. Zuidema,et al.  Simple rules can explain discrimination of putative recursive syntactic structures by a songbird species , 2009, Proceedings of the National Academy of Sciences.

[77]  Panayotis Mavromatis,et al.  Minimum description length modelling of musical structure , 2009 .

[78]  Remco C. Veltkamp,et al.  Modeling Harmonic Similarity Using a Generative Grammar of Tonal Harmony , 2009, ISMIR.

[79]  P. Mitra,et al.  De novo establishment of wild-type song culture in the zebra finch , 2009, Nature.

[80]  F. A. Gore Ouseley,et al.  A Treatise on Harmony , 2008 .

[81]  Walter T. Herbranson,et al.  Artificial grammar learning in pigeons , 2008, Learning & behavior.

[82]  C. J. Clark,et al.  The Anna's hummingbird chirps with its tail: a new mechanism of sonation in birds , 2008, Proceedings of the Royal Society B: Biological Sciences.

[83]  Clifton Callender,et al.  Generalized Voice-Leading Spaces , 2008, Science.

[84]  Esther Mondragón,et al.  Rule Learning by Rats , 2008, Science.

[85]  Michael C. Corballis,et al.  Recursion, Language, and Starlings , 2007, Cogn. Sci..

[86]  Carol L. Krumhansl,et al.  Modeling Tonal Tension , 2007 .

[87]  Dan Klein,et al.  Improved Inference for Unlexicalized Parsing , 2007, NAACL.

[88]  P. Grünwald The Minimum Description Length Principle (Adaptive Computation and Machine Learning) , 2007 .

[89]  M. Yip The search for phonology in other species , 2006, Trends in Cognitive Sciences.

[90]  Dmitri Tymoczko,et al.  The Geometry of Musical Chords , 2006, Science.

[91]  R. Jackendoff,et al.  The capacity for music: What is it, and what’s special about it? , 2006, Cognition.

[92]  W. Fitch The biology and evolution of music: A comparative perspective , 2006, Cognition.

[93]  Timothy Q. Gentner,et al.  Recursive syntactic pattern learning by songbirds , 2006, Nature.

[94]  David Huron Sweet Anticipation: Music and the Psychology of Expectation , 2006 .

[95]  Marcus T. Pearce,et al.  The construction and evaluation of statistical models of melodic structure in music perception and composition , 2005 .

[96]  Z. Dienes,et al.  Implicit learning of nonlocal musical rules: implicitly learning more than chunks. , 2005, Journal of experimental psychology. Learning, memory, and cognition.

[97]  Antonella Sorace,et al.  Gradience in Linguistic Data , 2005 .

[98]  Jean-Luc Gauvain,et al.  Training Neural Network Language Models on Very Large Corpora , 2005, HLT.

[99]  Kevin Knight,et al.  An Overview of Probabilistic Tree Transducers for Natural Language Processing , 2005, CICLing.

[100]  Geraint A. Wiggins,et al.  Improved Methods for Statistical Modelling of Monophonic Music , 2004 .

[101]  C. Krumhansl The Cognition of Tonality – as We Know it Today , 2004 .

[102]  Christopher Raphael,et al.  Functional Harmonic Analysis Using Probabilistic Models , 2004, Computer Music Journal.

[103]  K. Okanoya The Bengalese Finch: A Window on the Behavioral Neurobiology of Birdsong Syntax , 2004, Annals of the New York Academy of Sciences.

[104]  Noam Chomsky,et al.  The faculty of language: what is it, who has it, and how did it evolve? , 2002, Science.

[105]  F. Goller,et al.  Respiratory units of motor production and song imitation in the zebra finch. , 2002, Journal of neurobiology.

[106]  D. Temperley The Cognition of Basic Musical Structures , 2001 .

[107]  Paul Rodríguez,et al.  Simple Recurrent Networks Learn Context-Free and Context-Sensitive Languages by Counting , 2001, Neural Computation.

[108]  P. Kantor Foundations of Statistical Natural Language Processing , 2001, Information Retrieval.

[109]  T. Brants TnT – A Statistical Part-of-Speech Tagger , 2000, ANLP.

[110]  Chris Mellish,et al.  Statistical Learning of Harmonic Movement , 1999 .

[111]  E. Gibson,et al.  Memory Limitations and Structural Forgetting: The Perception of Complex Ungrammatical Sentences as Grammatical , 1999 .

[112]  Henrike Hultsch,et al.  How songbirds deal with large amounts of serial information: retrieval rules suggest a hierarchical song memory , 1998, Biological Cybernetics.

[113]  S. Hochreiter,et al.  Long Short-Term Memory , 1997, Neural Computation.

[114]  E. Schellenberg,et al.  Simplifying the Implication-Realization Model of Melodic Expectancy , 1997 .

[115]  R. Zann The Zebra Finch: A Synthesis of Field and Laboratory Studies , 1996 .

[116]  E. Schellenberg,et al.  Expectancy in melody: tests of the implication-realization model , 1996, Cognition.

[117]  Ian H. Witten,et al.  Multiple viewpoint systems for music prediction , 1995 .

[118]  C. Krumhansl Music Psychology and Music Theory: Problems and Prospects , 1995 .

[119]  R. Horton Rules and representations , 1993, The Lancet.

[120]  E. Narmour The Analysis and Cognition of Melodic Complexity: The Implication-Realization Model , 1992 .

[121]  Hava T. Siegelmann,et al.  On the computational power of neural nets , 1992, COLT '92.

[122]  Biing-Hwang Juang,et al.  Hidden Markov Models for Speech Recognition , 1991 .

[123]  Mark Derthick,et al.  Connections and symbols , 1990 .

[124]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[125]  J. Cynx,et al.  Experimental determination of a unit of song production in the zebra finch (Taeniopygia guttata). , 1990, Journal of comparative psychology.

[126]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[127]  J. Fodor,et al.  Connectionism and cognitive architecture: A critical analysis , 1988, Cognition.

[128]  James L. McClelland,et al.  PDP models and general issues in cognitive science , 1986 .

[129]  Stuart M. Shieber,et al.  Evidence against the context-freeness of natural language , 1985 .

[130]  Christopher Culy,et al.  The complexity of the vocabulary of Bambara , 1985 .

[131]  Robert B. Cantrick,et al.  A Generative Theory of Tonal Music , 1985 .

[132]  Mark Steedman,et al.  A Generative Grammar for Jazz Chord Sequences , 1984 .

[133]  P. Marler,et al.  Species-universal microstructure in the learned song of the swamp sparrow (Melospiza georgiana) , 1984, Animal Behaviour.

[134]  Allan Keiler,et al.  On Some Properties of Schenker's Pitch Derivations , 1983 .

[135]  P. J. B. Slater,et al.  Sequences of song in chaffinches , 1983, Animal Behaviour.

[136]  C. Krumhansl,et al.  Tracing the dynamic changes in perceived tonal organization in a spatial representation of musical keys. , 1982, Psychological review.

[137]  László Dezsö,et al.  Universal Grammar , 1981, Certainty in Action.

[138]  D. Hofstadter,et al.  Godel, Escher, Bach: An Eternal Golden Braid , 1979 .

[139]  Martin Kay,et al.  Syntactic Process , 1979, ACL.

[140]  Allan KElLER,et al.  BERNSTEIN'S THE UNANSWERED QUESTION AND THE PROBLEM OF MUSICAL COMPETENCE , 1978 .

[141]  Dietmar Todt,et al.  Social Learning of Vocal Patterns and Modes of their Application in Grey Parrots (Psittacus erithacus)1,2,3 , 1975 .

[142]  R. Payne,et al.  Songs of Humpback Whales , 1971, Science.

[143]  Erwin R. Jacobi,et al.  Treatise on Harmony , 1971 .

[144]  Terry Winograd,et al.  Linguistics and the computer analysis of tonal harmony , 1968 .

[145]  Walter S. Stolz,et al.  A study of the ability to decode grammatically novel sentences , 1967 .

[146]  A. Reber Implicit learning of artificial grammars , 1967 .

[147]  A M Liberman,et al.  Perception of the speech code. , 1967, Psychological review.

[148]  George A. Miller,et al.  Free Recall of Self-Embedded English Sentences , 1964, Inf. Control..

[149]  M. Zengel Literacy as a Factor in Language Change , 1962 .

[150]  Alaa A. Kharbouch,et al.  Three models for the description of language , 1956, IRE Trans. Inf. Theory.

[151]  G. Mckay Harmony , 1955, Journalen sykepleien.

[152]  R. Paget The Origin of Speech , 1927, Nature.

[153]  Martin Rohrmeier,et al.  Towards a Syntax of the Classical Cadence , 2015 .

[154]  Tillman Weyde,et al.  Hybrid Long- and Short-Term Models of Folk Melodies , 2015, ISMIR.

[155]  M. Knörnschild,et al.  Male courtship displays and vocal communication in the polygynous bat Carollia perspicillata , 2014 .

[156]  Luc Steels,et al.  Language in the light of evolution , 2014 .

[157]  Martin Rohrmeier,et al.  Implicit Learning and Recursion , 2014 .

[158]  Jiani Chen,et al.  Artificial grammar learning in zebra finches and human adults: XYX vs XXY , 2014 .

[159]  Willem H. Zuidema Context-freeness Revisited , 2013, CogSci.

[160]  Thore Graepel,et al.  Comparing Feature-Based Models of Harmony , 2012 .

[161]  Willem H. Zuidema Language in Nature: on the Evolutionary Roots of a Cultural Phenomenon (draft chapter for The Language Phenomenon) , 2012 .

[162]  Vysoké Učení,et al.  Statistical Language Models Based on Neural Networks , 2012 .

[163]  H. Taylor,et al.  The Australian pied butcherbird and the natureculture continuum , 2011 .

[164]  Christopher D. Manning,et al.  Learning Continuous Phrase Representations and Syntactic Parsing with Recursive Neural Networks , 2010 .

[165]  Lukás Burget,et al.  Recurrent neural network based language model , 2010, INTERSPEECH.

[166]  J. Rissanen Minimum Description Length Principle. , 2010 .

[167]  Shravan Vasishth,et al.  Processing grammatical and ungrammatical center embeddings in English and German: A computational model , 2009 .

[168]  I. Cross,et al.  Music as a communicative medium , 2009 .

[169]  Mark Steedman The Blues and the Abstract Truth: Music and Mental Models , 2009 .

[170]  Ian Cross,et al.  Statistical Properties of Tonal Harmony in Bach ’ s Chorales , 2008 .

[171]  Jean-François Paiement,et al.  Probabilistic models for music , 2008 .

[172]  E. Pothos Theories of artificial grammar learning. , 2007, Psychological bulletin.

[173]  M. Rohrmeier A generative grammar approach to diatonic harmonic structure , 2007 .

[174]  Panayotis Mavromatis,et al.  A Hidden Markov Model of Melody Production in greek Church Chant , 2006 .

[175]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[176]  B. Aarts Modelling linguistic gradience , 2004 .

[177]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[178]  Robert O. Gjerdingen,et al.  The Cognition of Basic Musical Structures , 2004 .

[179]  Tuomas Eerola,et al.  The dynamics of musical expectancy : cross-cultural and statistical approaches to melodic expectations , 2003 .

[180]  Wojciech Skut,et al.  SYNTACTIC ANNOTATION OF A GERMAN NEWSPAPER CORPUS , 2003 .

[181]  Stuart J. Russell,et al.  Dynamic bayesian networks: representation, inference and learning , 2002 .

[182]  Steven Abney,et al.  Statistical Methods and Linguistics , 2002 .

[183]  P. Kuhl,et al.  Birdsong and human speech: common themes and mechanisms. , 1999, Annual review of neuroscience.

[184]  Kevin P. Murphy Information theory , 1998 .

[185]  Schellenberg Eg Expectancy in melody: tests of the implication-realization model , 1996 .

[186]  Prof. Homayoon Beigi,et al.  Fundamentals of speech recognition , 1993, Prentice Hall signal processing series.

[187]  David J. Weir,et al.  The convergence of mildly context-sensitive grammar formalisms , 1990 .

[188]  Jordan B. Pollack,et al.  Implications of Recursive Distributed Representations , 1988, NIPS.

[189]  A. Joshi,et al.  Natural language parsing: Tree adjoining grammars: How much context-sensitivity is required to provide reasonable structural descriptions? , 1985 .

[190]  Peter Marler,et al.  Ordering of sequences of singing behaviour of mistle thrushes in relationship to timing , 1963 .

[191]  Lucien Tesnière Éléments de syntaxe structurale , 1959 .

[192]  Heinrich Schenker,et al.  Der freie Satz , 1935 .

[193]  W. Fitch,et al.  References and Notes Materials and Methods Figs. S1 to S6 Tables S1 and S2 References Computational Constraints on Syntactic Processing in a Nonhuman Primate , 2022 .