Musical Syntax II: Empirical Perspectives

Efforts to develop a formal characterization of musical structure are often framed in syntactic terms, sometimes but not always with direct inspiration from research on language. In Chap. 25, we present syntactic approaches to characterizing musical structure and survey a range of theoretical issues involved in developing formal syntactic theories of sequential structure in music. Such theories are often computational in nature, lending themselves to implementation and our first goal here is to review empirical research on computational modeling of musical structure from a syntactic point of view. We ask about the motivations for implementing a model and assess the range of approaches that have been taken to date. It is important to note that while a computational model may be capable of deriving an optimal structural description of a piece of music, human cognitive processing may not achieve this optimal performance, or may even process syntax in a different way. Therefore we emphasize the difference between developing an optimal model of syntactic processing and developing a model that simulates human syntactic processing. Furthermore, we argue that, while optimal models (e. g., optimal compression or prediction) can be useful as a benchmark or yardstick for assessing human performance, if we wish to understand human cognition then simulating human performance (including aspects that are nonoptimal or even erroneous) should be the priority. Following this principle, we survey research on processing of musical syntax from the perspective of computational modeling, experimental psychology and cognitive neuroscience. There exists a large number of computational models of musical syntax, but we limit ourselves to those that are explicitly cognitively motivated, assessing them in the context of theoretical, psychological and neuroscientific research.

[1]  C C Wood,et al.  Event-related potentials elicited by deviant endings to melodies. , 1992, Psychophysiology.

[2]  Vladimir J. Konečni,et al.  The Effects of Instrumentation, Playing Style, and Structure in the Goldberg Variations by Johann Sebastian Bach , 1985 .

[3]  Angela D. Friederici,et al.  Brain potentials indicate immediate use of prosodic cues in natural speech processing , 1999, Nature Neuroscience.

[4]  Stefan Koelsch,et al.  Differences in Electric Brain Responses to Melodies and Chords , 2010, Journal of Cognitive Neuroscience.

[5]  S. Koelsch,et al.  Predictive information processing in music cognition. A critical review. , 2012, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[6]  Marcus T. Pearce,et al.  Melodic pitch expectation interacts with neural responses to syntactic but not semantic violations , 2013, Cortex.

[7]  Jordan B. Pollack,et al.  Reduced Memory Representations for Music , 1995, Cogn. Sci..

[8]  Mireille Besson,et al.  Visually Induced Auditory Expectancy in Music Reading: A Behavioral and Electrophysiological Study , 2005, Journal of Cognitive Neuroscience.

[9]  Mark Steedman,et al.  A Robust Parser-Interpreter for Jazz Chord Sequences , 2014 .

[10]  R. Shepard,et al.  Tonal Schemata in the Perception of Music in Bali and in the West , 1984 .

[11]  M. Pearce,et al.  Predictive uncertainty in auditory sequence processing , 2014, Front. Psychol..

[12]  Christopher Raphael,et al.  Functional Harmonic Analysis Using Probabilistic Models , 2004, Computer Music Journal.

[13]  E. Narmour The Analysis and Cognition of Melodic Complexity: The Implication-Realization Model , 1992 .

[14]  A. Friederici,et al.  Brain Indices of Music Processing: Nonmusicians are Musical , 2000, Journal of Cognitive Neuroscience.

[15]  Satoshi Tojo,et al.  Implementing “A Generative Theory of Tonal Music” , 2006 .

[16]  Ray Jackendoff,et al.  Musical Parsing and Musical Affect , 1991 .

[17]  Alan Marsden,et al.  Schenkerian Analysis by Computer: A Proof of Concept , 2010 .

[18]  I. Cross,et al.  Artificial Grammar Learning of Melody Is Constrained by Melodic Inconsistency: Narmour's Principles Affect Melodic Learning , 2013, PloS one.

[19]  Noam Chomsky,et al.  वाक्यविन्यास का सैद्धान्तिक पक्ष = Aspects of the theory of syntax , 1965 .

[20]  Seung-Goo Kim,et al.  The Effect of Conditional Probability of Chord Progression on Brain Response: An MEG Study , 2011, PloS one.

[21]  C. Krumhansl Part I: Ethology/Evolution—Do Animals Have Music or Something Else?: Introduction , 2005 .

[22]  Geraint A. Wiggins,et al.  Methods for Combining Statistical Models of Music , 2004, CMMR.

[23]  M. Pearce,et al.  Tracking of pitch probabilities in congenital amusia , 2012, Neuropsychologia.

[24]  Von Hippel,et al.  Melodic-Expectation Rules as Learned Heuristics , 2002 .

[25]  J. Bharucha Music Cognition and Perceptual Facilitation: A Connectionist Framework , 1987 .

[26]  L. Baum,et al.  Statistical Inference for Probabilistic Functions of Finite State Markov Chains , 1966 .

[27]  H. Riemann Musikalische Syntaxis : Grundriß einer harmonischen Satzbildungslehre , 1877 .

[28]  Alan Marsden,et al.  Generative structural representation of tonal music , 2005 .

[29]  Paul T. von Hippel,et al.  Why Do Skips Precede Reversals? The Effect of Tessitura on Melodic Structure , 2000 .

[30]  Stefan Koelsch,et al.  Shared neural resources between music and language indicate semantic processing of musical tension-resolution patterns. , 2008, Cerebral cortex.

[31]  Barbara Tillmann,et al.  Implicit learning of tonality: A self-organizing approach , 2000 .

[32]  R. Poldrack Can cognitive processes be inferred from neuroimaging data? , 2006, Trends in Cognitive Sciences.

[33]  C. Krumhansl,et al.  Melodic Expectation in Finnish Spiritual Folk Hymns: Convergence of Statistical, Behavioral, and Computational Approaches , 1999 .

[34]  T. Gunter,et al.  Music matters: preattentive musicality of the human brain. , 2002, Psychophysiology.

[35]  M. Woldorff,et al.  Effects of attention on the neural processing of harmonic syntax in Western music. , 2005, Brain research. Cognitive brain research.

[36]  M. Pearce,et al.  Electrophysiological correlates of melodic processing in congenital amusia , 2013, Neuropsychologia.

[37]  Allen Newell,et al.  Computer science as empirical inquiry: symbols and search , 1976, CACM.

[38]  R. Shepard 11 – Structural Representations of Musical Pitch , 1982 .

[39]  Panayotis Mavromatis,et al.  Exploring the Rhythm of the Palestrina Style A Case Study in Probabilistic Grammar Induction , 2012 .

[40]  S. Koelsch Towards a neural basis of processing musical semantics. , 2011, Physics of life reviews.

[41]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[42]  James C. Carlsen Some factors which influence melodic expectancy , 1981 .

[43]  Martin Rohrmeier,et al.  Towards a generative syntax of tonal harmony , 2011 .

[44]  Mark Steedman The Blues and the Abstract Truth: Music and Mental Models , 2009 .

[45]  Yoshua Bengio,et al.  Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription , 2012, ICML.

[46]  Jean-François Paiement,et al.  Predictive models for music , 2009, Connect. Sci..

[47]  I. Peretz,et al.  The amusic brain: in tune, out of key, and unaware. , 2009, Brain : a journal of neurology.

[48]  David Marr,et al.  VISION A Computational Investigation into the Human Representation and Processing of Visual Information , 2009 .

[49]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[50]  S M Kosslyn,et al.  If neuroimaging is the answer, what is the question? , 1999, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[51]  Craig A. Kaplan,et al.  Foundations of cognitive science , 1989 .

[52]  Barbara Tillmann,et al.  Tonal Expectations Influence Early Pitch Processing , 2011, Journal of Cognitive Neuroscience.

[53]  A D Friederici,et al.  Prosodic Boundaries, Comma Rules, and Brain Responses: The Closure Positive Shift in ERPs as a Universal Marker for Prosodic Phrasing in Listeners and Readers , 2001, Journal of psycholinguistic research.

[54]  E. Schellenberg,et al.  Simplifying the Implication-Realization Model of Melodic Expectancy , 1997 .

[55]  Schellenberg Eg Expectancy in melody: tests of the implication-realization model , 1996 .

[56]  Stephen W. Smoliar,et al.  A computer aid for Schenkerian analysis , 1980, ACM '79.

[57]  Stefan Koelsch,et al.  Interaction between Syntax Processing in Language and in Music: An ERP Study , 2005, Journal of Cognitive Neuroscience.

[58]  Martin Rohrmeier,et al.  Implicit Learning and Acquisition of Music , 2012, Top. Cogn. Sci..

[59]  Alan Marsden,et al.  Representing Melodic Patterns as Networks of Elaborations , 2001, Comput. Humanit..

[60]  C. Krumhansl,et al.  Tonal hierarchies in the music of north India. , 1984, Journal of experimental psychology. General.

[61]  B Tillmann,et al.  The effect of harmonic context on phoneme monitoring in vocal music , 2001, Cognition.

[62]  Mark Steedman,et al.  The syntactic process , 2004, Language, speech, and communication.

[63]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[64]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[65]  R. Jackendoff,et al.  A Generative Theory of Tonal Music , 1985 .

[66]  Terry Winograd,et al.  Linguistics and the computer analysis of tonal harmony , 1968 .

[67]  Aniruddh D. Patel,et al.  Musical syntactic processing in agrammatic Broca's aphasia , 2008 .

[68]  Aniruddh D. Patel,et al.  Processing Syntactic Relations in Language and Music: An Event-Related Potential Study , 1998, Journal of Cognitive Neuroscience.

[69]  B. Tillmann,et al.  Effect of global structure and temporal organization on chord processing , 1999 .

[70]  Stephen W. Smoliar,et al.  Schenker's theory of tonal music—its explication through computational processes , 1978 .

[71]  Allen Forte,et al.  Introduction to Schenkerian Analysis , 1984 .

[72]  Kemal Ebcioglu,et al.  An Expert System for Harmonizing Four-Part Chorales , 1988, ICMC.

[73]  E. Narmour,et al.  Beyond Schenkerism: The Need for Alternatives in Music Analysis , 1978 .

[74]  Aniruddh D. Patel Why would Musical Training Benefit the Neural Encoding of Speech? The OPERA Hypothesis , 2011, Front. Psychology.

[75]  S. Koelsch,et al.  Effects of Unexpected Chords and of Performer's Expression on Brain Responses and Electrodermal Activity , 2008, PloS one.

[76]  Eugene Narmour,et al.  The Analysis and Cognition of Basic Melodic Structures: The Implication-Realization Model , 1990 .

[77]  Joel E. Cohen,et al.  Information theory and music , 2007 .

[78]  I. Cross,et al.  Incidental and online learning of melodic structure , 2011, Consciousness and Cognition.

[79]  Michael C. Mozer,et al.  Neural Network Music Composition by Prediction: Exploring the Benefits of Psychoacoustic Constraints and Multi-scale Processing , 1994, Connect. Sci..

[80]  Carol L. Krumhansl,et al.  Modeling Tonal Tension , 2007 .

[81]  Marcus T. Pearce,et al.  The construction and evaluation of statistical models of melodic structure in music perception and composition , 2005 .

[82]  Remco C. Veltkamp,et al.  Modeling Harmonic Similarity Using a Generative Grammar of Tonal Harmony , 2009, ISMIR.

[83]  Ramon Fuller Structure and Information in Webern's Symphonie, Op. 21 , 1967 .

[84]  Ian H. Witten,et al.  Multiple viewpoint systems for music prediction , 1995 .

[85]  Michael T. Ullman,et al.  Double dissociation between rules and memory in music: An event-related potential study , 2007, NeuroImage.

[86]  Uli Reich,et al.  The meanings of semantics: Comment on "Towards a neural basis of processing musical semantics" by Stefan Koelsch. , 2011, Physics of life reviews.

[87]  B. Tillmann,et al.  Repetition priming: Is music special? , 2005, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[88]  David Jensen,et al.  Using supervised learning to uncover deep musical structure , 2015, AAAI 2015.

[89]  L. Cuddy,et al.  Responsiveness of Western adults to pitch-distributional information in melodic sequences , 1995, Psychological research.

[90]  Angela D. Friederici,et al.  Prosody-driven Sentence Processing: An Event-related Brain Potential Study , 2005, Journal of Cognitive Neuroscience.

[91]  Panayotis Mavromatis,et al.  Minimum description length modelling of musical structure , 2009 .

[92]  H. C. Longuet-Higgins Artificial intelligence — a new theroretical psychology? , 1981, Cognition.

[93]  Russell A Poldrack,et al.  Using fMRI to Constrain Theories of Cognition , 2013, Perspectives on psychological science : a journal of the Association for Psychological Science.

[94]  M. Besson,et al.  AN EVENT-RELATED POTENTIAL (ERP) STUDY OF MUSICAL EXPECTANCY : COMPARISON OF MUSICIANS WITH NONMUSICIANS , 1995 .

[95]  Mari Riess Jones,et al.  Does rule recursion make melodies easier to reproduce? If not, what does? , 1986, Cognitive Psychology.

[96]  Stefan Koelsch,et al.  Processing of hierarchical syntactic structure in music , 2013, Proceedings of the National Academy of Sciences.

[97]  Phillip B. Kirlin Using Harmonic and Melodic Analyses to Automate the Initial Stages of Schenkerian Analysis , 2009, ISMIR.

[98]  William Earl Caplin,et al.  Classical Form: A Theory of Formal Functions for the Instrumental Music of Haydn, Mozart, and Beethoven , 1998 .

[99]  Charles Ames,et al.  The Markov Process as a Compositional Model: A Survey and Tutorial , 2017 .

[100]  María Herrojo Ruiz,et al.  Unsupervised statistical learning underpins computational, behavioural, and neural manifestations of musical expectation , 2010, NeuroImage.

[101]  D. Deutsch,et al.  The Internal Representation of Pitch Sequences in Tonal Music , 1981 .

[102]  Angela D. Friederici,et al.  Effects of Musical Expertise and Boundary Markers on Phrase Perception in Music , 2006 .

[103]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[104]  Nicholas Cook The Perception of Large-Scale Tonal Closure , 1987 .

[105]  Juan Pablo Bello,et al.  A Robust Mid-Level Representation for Harmonic Content in Music Signals , 2005, ISMIR.

[106]  J. Youngblood Style as Information , 1958 .

[107]  Nancy Kanwisher,et al.  How fMRI Can Inform Cognitive Theories , 2013, Perspectives on psychological science : a journal of the Association for Psychological Science.

[108]  J. Bharucha,et al.  Reaction time and musical expectancy: priming of chords. , 1986, Journal of experimental psychology. Human perception and performance.

[109]  John G. Cleary,et al.  MODELLING AND GENERATING MUSIC USING MULTIPLE VIEWPOINTS , 1988 .

[110]  Frank A. Russo,et al.  The motor origins of human and avian song structure , 2011, Proceedings of the National Academy of Sciences.

[111]  Charles Ames,et al.  Automated Composition in Retrospect: 1956–1986 , 2017 .

[112]  William Forde Thompson,et al.  Expectancy in Bohemian Folk Song Melodies: Evaluation of Implicative Principles for Implicative and Closural Intervals , 1998 .

[113]  M. Kassler APL applied in music theory , 1987 .

[114]  Martin Rohrmeier,et al.  Incidental Learning of Melodic Structure of North Indian Music , 2017, Cogn. Sci..

[115]  Panayotis Mavromatis,et al.  HMM Analysis of Musical Structure: Identification of Latent Variables Through Topology-Sensitive Model Selection , 2009 .

[116]  Stefan Koelsch,et al.  Bach Speaks: A Cortical “Language-Network” Serves the Processing of Music , 2002, NeuroImage.

[117]  John R. Anderson,et al.  Learning Artificial Grammars With Competitive Chunking , 1990 .

[118]  B. Lindblom,et al.  Generative theories in language and music descriptions , 1976, Cognition.

[119]  Mayumi Adachi,et al.  Expectancy in melody: tests of children and adults. , 2002, Journal of experimental psychology. General.

[120]  V. Konečni Elusive Effects of Artists' ‘Messages’ , 1984 .

[121]  Vladimir J. Konečni,et al.  The Effects of Structural Interventions in the First Movement of Mozart's Symphony in G Minor K. 550 on Aesthetic Preference , 1992 .

[122]  T. Eerola Data-driven influences on melodic expectancy : Continuations in North Sami yoiks rated by South African traditional healers , 2004 .

[123]  William Drabkin,et al.  THE CONCEPT OF MUSICAL GRAMMAR , 1983 .

[124]  Hiroshi Nittono,et al.  Event-related potentials elicited by wrong terminal notes: effects of temporal disruption , 2000, Biological Psychology.

[125]  H. V. D. van der Lely,et al.  Electrical Brain Responses in Language-Impaired Children Reveal Grammar-Specific Deficits , 2008, PloS one.

[126]  A. Friederici Towards a neural basis of auditory sentence processing , 2002, Trends in Cognitive Sciences.

[127]  Geraint A. Wiggins,et al.  Auditory Expectation: The Information Dynamics of Music Perception and Cognition , 2012, Top. Cogn. Sci..

[128]  Satoshi Tojo,et al.  Fatta: Full Automatic Time-Span Tree Analyzer , 2007, ICMC.

[129]  Phillip B. Kirlin A Data Set for Computational Studies of Schenkerian Analysis , 2014, ISMIR.

[130]  Geraint A. Wiggins,et al.  Multiple Viewpoint Systems: Time Complexity and the Construction of Domains for Complex Musical Viewpoints in the Harmonization Problem , 2013 .

[131]  Geraint A. Wiggins,et al.  Probabilistic models of expectation violation predict psychophysiological emotional responses to live concert music , 2013, Cognitive, Affective, & Behavioral Neuroscience.

[132]  Leonard B. Meyer Meaning in music and information theory. , 1957 .

[133]  Vinod Menon,et al.  Musical structure is processed in “language” areas of the brain: a possible role for Brodmann Area 47 in temporal coherence , 2003, NeuroImage.

[134]  M. Rohrmeier A generative grammar approach to diatonic harmonic structure , 2007 .

[135]  W. B. Haas Music information retrieval based on tonal harmony , 2012 .

[136]  Frank A. Russo,et al.  A common origin for vocal accuracy and melodic expectancy: Vocal constraints , 1999 .

[137]  Ian H. Witten,et al.  Comparing human and computational models of music prediction , 1994 .

[138]  S. Koelsch,et al.  Effects of musical expertise on the early right anterior negativity: an event-related brain potential study. , 2002, Psychophysiology.

[139]  R. Knight,et al.  A Generalized Mechanism for Perception of Pitch Patterns , 2009, The Journal of Neuroscience.

[140]  Justyna Humięcka-Jakubowska,et al.  Sweet Anticipation : Music and , 2006 .

[141]  R Verleger,et al.  P3-evoking wrong notes: unexpected, awaited, or arousing? , 1990, The International journal of neuroscience.

[142]  Robert O. Gjerdingen An Experimental Music Theory , 1999 .

[143]  Marcus T. Pearce,et al.  Music Cognition and the Cognitive Sciences , 2012, Top. Cogn. Sci..

[144]  Frans Wiering,et al.  Improving Audio Chord Transcription by Exploiting Harmonic and Metric Knowledge , 2012, ISMIR.

[145]  P. Janata,et al.  Activation of the inferior frontal cortex in musical priming. , 2003, Annals of the New York Academy of Sciences.

[146]  Stefan Koelsch,et al.  The Role of Harmonic Expectancy Violations in Musical Emotions: Evidence from Subjective, Physiological, and Neural Responses , 2006, Journal of Cognitive Neuroscience.

[147]  D. Västfjäll,et al.  Emotional responses to music: the need to consider underlying mechanisms. , 2008, The Behavioral and brain sciences.

[148]  Jordan B. Pollack,et al.  Recursive Distributed Representations , 1990, Artif. Intell..

[149]  Ian Cross,et al.  Tacit tonality - Implicit learning of context-free harmonic structure , 2009 .

[150]  E. Vincent,et al.  Melody harmonisation with interpolated probabilistic models , 2012 .

[151]  Daniele Schön,et al.  Cerebral Cortex doi:10.1093/cercor/bhr022 Musical Expertise Boosts Implicit Learning of Both Musical and Linguistic Structures , 2011 .

[152]  L. Baum,et al.  A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains , 1970 .

[153]  M. Tervaniemi,et al.  Representation of harmony rules in the human brain: Further evidence from event-related potentials , 2007, Brain Research.

[154]  C. Bean Information Theory Analyses of Four Sonata Expositions , 1966 .

[155]  Alexander R. Brinkman,et al.  The Effect of Modulation and Formal Manipulation on Perception of Tonic Closure by Expert Listeners , 1999 .

[156]  B. Tillmann Implicit Investigations of Tonal Knowledge in Nonmusician Listeners , 2005, Annals of the New York Academy of Sciences.

[157]  Matthew Brown,et al.  PARSING CONTEXT-FREE GRAMMARS FOR MUSIC: A COMPUTATIONAL MODEL OF SCHENKERIAN ANALYSIS , 2004 .

[158]  B. Tillmann,et al.  Sensory versus cognitive components in harmonic priming. , 2003, Journal of experimental psychology. Human perception and performance.

[159]  Elizabeth K. Johnson,et al.  Statistical learning of tone sequences by human infants and adults , 1999, Cognition.

[160]  Tillman Weyde,et al.  Learning Distributed Representations for Multiple-Viewpoint Melodic Prediction , 2013 .

[161]  Michael J. Martinez,et al.  Music and language side by side in the brain: a PET study of the generation of melodies and sentences , 2006, The European journal of neuroscience.

[162]  A. Friederici,et al.  Musical syntax is processed in Broca's area: an MEG study , 2001, Nature Neuroscience.

[163]  Richard C. Pinkerton Information theory and melody. , 1956 .

[164]  Alexander Clark,et al.  Learning trees from strings: a strong learning algorithm for some context-free grammars , 2013, J. Mach. Learn. Res..

[165]  Mark Steedman,et al.  A Generative Grammar for Jazz Chord Sequences , 1984 .

[166]  Stephen W. Smoliar,et al.  A LISP-based system for the study of Schenkerian analysis , 1976 .

[167]  Stefan Koelsch,et al.  Children with Specific Language Impairment Also Show Impairment of Music-syntactic Processing , 2008, Journal of Cognitive Neuroscience.

[168]  C. Krumhansl Music Psychology and Music Theory: Problems and Prospects , 1995 .

[169]  C. Krumhansl,et al.  Cross-cultural music cognition: cognitive methodology applied to North Sami yoiks , 2000, Cognition.

[170]  Rens Bod,et al.  Memory-Based Models of Melodic Analysis: Challenging the Gestalt Principles , 2002 .

[171]  O. Witte,et al.  Perception of phrase structure in music , 2005, Human brain mapping.

[172]  D Deutsch,et al.  The processing of structured and unstructured tonal sequences , 1980, Perception & psychophysics.

[173]  J. Bharucha,et al.  Priming of chords: Spreading activation or overlapping frequency spectra? , 1987, Perception & psychophysics.

[174]  Panayotis Mavromatis,et al.  A Hidden Markov Model of Melody Production in greek Church Chant , 2006 .