Toward a computational model of expression in music performance: The GERM model

This article presents a computational model of expression in music performance: the GERM model. The purpose of the GERM model is to (a) describe the principal sources of variability in music performance, (b) emphasize the need to integrate different aspects of performance in a common model, and (c) provide some preliminaries (germ = a basis from which a thing may develop) for a computational model that simulates the different aspects. Drawing on previous research on performance, we propose that performance expression derives from four main sources of variability: (1) Generative Rules, which function to convey the generative structure in a musical manner (e.g., Clarke, 1988; Sundberg, 1988); (2) Emotional Expression, which is governed by the performer's expressive intention (e.g., Juslin, 1997a); (3) Random Variations, which reflect internal timekeeper variance and motor delay variance (e.g., Gilden, 2001; Wing and Kristofferson, 1973); and (4) Movement Principles, which prescribe that certain features of the performance are shaped in accordance with biological motion (e.g., Shove and Repp, 1995). A preliminary version of the GERM model was implemented by means of computer synthesis. Synthesized performances were evaluated by musically trained participants in a listening test. The results from the test support a decomposition of expression in terms of the GERM model. Implications for future research on music performance are discussed.

[1]  J. Stainer,et al.  The Emotions , 1922, Nature.

[2]  Alexander Truslit Gestaltung und Bewegung in der Musik , 1938 .

[3]  Irwin Pollack,et al.  Communication of Verbal Modes of Expression , 1960 .

[4]  W. J. McGill,et al.  Random fluctuations of response rate , 1962 .

[5]  A. Kristofferson,et al.  Response delays and the timing of discrete motor responses , 1973 .

[6]  G. Johansson Visual perception of biological motion and a model for its analysis , 1973 .

[7]  G. M. Robinson,et al.  Rhythm is processed by the speech hemisphere. , 1974, Journal of experimental psychology.

[8]  T. Bever,et al.  Cerebral Dominance in Musicians and Nonmusicians , 1974, Science.

[9]  M. Natale,et al.  Perception of nonlinguistic auditory rhythms by the speech hemisphere , 1977, Brain and Language.

[10]  J. Bradshaw,et al.  The role of the cerebral hemispheres in music , 1977, Brain and Language.

[11]  H. W. Gordon Left Hemisphere Dominance for Rhythmic Elements in Dichotically-Presented Melodies , 1978, Cortex.

[12]  M. Mesulam,et al.  Dominant language functions of the right hemisphere? Prosody and emotional gesturing. , 1979, Archives of neurology.

[13]  Ruth Rasch,et al.  Synchronization in performed ensemble music , 1979 .

[14]  J. Singer,et al.  Cardiovascular Differentiation of Happiness, Sadness, Anger, and Fear Following Imagery and Exercise1 , 1981, Psychosomatic medicine.

[15]  M. P. Bryden,et al.  A left-ear advantage for identifying the emotional quality of tonal sequences , 1982, Neuropsychologia.

[16]  L. Shaffer Rhythm and timing in skill. , 1982, Psychological review.

[17]  Shaffer Lh Rhythm and timing in skill. , 1982 .

[18]  J. Sloboda The Communication of Musical Metre in Piano Performance , 1983 .

[19]  P. Ekman,et al.  Autonomic nervous system activity distinguishes among emotions. , 1983, Science.

[20]  Eugene Narmour,et al.  Some Major Theoretical Problems concerning the Concept of Hierarchy in the Analysis of Tonal Music , 1983 .

[21]  S. Hibi,et al.  Rhythm perception in repetitive sound sequence , 1983 .

[22]  J. Fodor The Modularity of mind. An essay on faculty psychology , 1986 .

[23]  S. Runeson,et al.  Kinematic specification of dynamics as an informational basis for person and action perception: Expe , 1983 .

[24]  Robert J. Zatorre,et al.  Musical Perception and Cerebral Function: A Critical Review , 1984 .

[25]  S. Keele,et al.  Motor Disorder and the Timing of Repetitive Movements a , 1984, Annals of the New York Academy of Sciences.

[26]  Sverker Runeson,et al.  Perceiving people through their movements , 1985 .

[27]  J. Sloboda Generative Processes in Music: The Psychology of Performance, Improvisation, and Composition , 1985 .

[28]  Barbara Shapiro,et al.  The role of the right hemisphere in the control of speech prosody in propositional and affective contexts , 1985, Brain and Language.

[29]  T. Flash,et al.  The coordination of arm movements: an experimentally confirmed mathematical model , 1985, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[30]  N. Todd A Model of Expressive Timing in Tonal Music , 1985 .

[31]  C. Atkeson,et al.  Kinematic features of unrestrained vertical arm movements , 1985, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[32]  D. Ploog BIOLOGICAL FOUNDATIONS OF THE VOCAL EXPRESSIONS OF EMOTIONS , 1986 .

[33]  John A. Sloboda,et al.  The performance of music , 1986 .

[34]  Steven W Keele Sequencing and Timing in Skilled Perception and Action: An Overview. , 1986 .

[35]  K. Scherer Vocal affect expression: a review and a model for future research. , 1986, Psychological bulletin.

[36]  Rolf Carlson,et al.  Speech and music performance: Parallels and contrasts , 1987 .

[37]  C. Krumhansl Music as Cognition. , 1987 .

[38]  Anders Friberg,et al.  Rules for automated performance of ensemble music , 1987 .

[39]  K. Emmorey,et al.  The neurological substrates for prosodic aspects of speech , 1987, Brain and Language.

[40]  P. Shaver,et al.  Emotion knowledge: further exploration of a prototype approach. , 1987, Journal of personality and social psychology.

[41]  M. Conway,et al.  Situational knowledge and emotions , 1987 .

[42]  M. Bryden,et al.  Dichotic laterality effects obtained with emotional words. , 1988 .

[43]  Alf Gabrielsson,et al.  Timing in music performance and its relations to music experience. , 1988 .

[44]  E. Clarke Generative principles in music performance. , 1988 .

[45]  S. Grossberg,et al.  Neural dynamics of planned arm movements: emergent invariants and speed-accuracy properties during trajectory formation. , 1988, Psychological review.

[46]  M. Peters The relationship between variability of intertap intervals and interval duration , 1989 .

[47]  C. Palmer Mapping musical thought to musical performance. , 1989, Journal of experimental psychology. Human perception and performance.

[48]  E. Clarke,et al.  The perception of expressive timing in music , 1989, Psychological research.

[49]  Bruno H. Repp,et al.  Further Perceptual Evaluations of Pulse Microstructure in Computer Performances of Classical Piano Music , 1990 .

[50]  I J Hirsh,et al.  Studies in auditory timing: 1. Simple patterns , 1990, Perception & psychophysics.

[51]  I. Peretz,et al.  Processing of local and global musical information by unilateral brain-damaged patients. , 1990, Brain : a journal of neurology.

[52]  B H Repp,et al.  Patterns of expressive timing in performances of a Beethoven minuet by nineteen famous pianists. , 1988, The Journal of the Acoustical Society of America.

[53]  Anders Friberg Generative Rules for Music Performance : A Formal Description of a Rule System , 1991 .

[54]  Johan Sundberg,et al.  Common Secrets of Musicians and Listeners - An analysis-by-synthesis Study of Musical Performance , 1991 .

[55]  L. Henry Shaffer How to interpret music. , 1992 .

[56]  John J. Magee,et al.  Categorical perception of facial expressions , 1992, Cognition.

[57]  B. Repp Probing the cognitive representation of musical time: Structural constraints on the perception of timing perturbations , 1992, Cognition.

[58]  J. Panksepp A critical role for "affective neuroscience" in resolving what is basic about basic emotions. , 1992, Psychological review.

[59]  John W. Moore,et al.  A Mechanism for Timing Conditioned Responses , 1992 .

[60]  Nell P. McAngusTodd,et al.  The dynamics of dynamics: A model of musical expression , 1992 .

[61]  Jacob Feldman,et al.  Force Dynamics of Tempo Change in Music , 1992 .

[62]  J Sergent,et al.  Distributed neural network underlying musical sight-reading and keyboard performance. , 1992, Science.

[63]  K. Oatley Best Laid Schemes: The Psychology of Emotions , 1992 .

[64]  P. Ekman An argument for basic emotions , 1992 .

[65]  M. Gluck,et al.  Explaining Basic Categories: Feature Predictability and Information , 1992 .

[66]  B. Repp Diversity and commonality in music performance: an analysis of timing microstructure in Schumann's "Träumerei". , 1992, The Journal of the Acoustical Society of America.

[67]  Peter van Oosten Critical study of Sundberg's rules for expression in the performance of melodies , 1993 .

[68]  D. Cicchetti Emotion and Adaptation , 1993 .

[69]  John D. C. Little,et al.  On model building , 1993 .

[70]  A. Muratorio,et al.  A case of music imperception. , 1993, Journal of neurology, neurosurgery, and psychiatry.

[71]  K. Heilman,et al.  Disruption of automatic speech following a right basal ganglia lesion , 1993, Neurology.

[72]  C. Drake,et al.  Tempo sensitivity in auditory sequences: Evidence for a multiple-look model , 1993, Perception & psychophysics.

[73]  Morris J. Cohen,et al.  Expressive Aprosodia Following Stroke to the Right Basal Ganglia: A Case Report , 1994 .

[74]  John A. Sloboda Music performance: Expression and the development of excellence. , 1994 .

[75]  Karen Donders,et al.  The detection of anisochrony in monaural and interaural sound sequences , 1994, Perception & psychophysics.

[76]  D L Gilden,et al.  1/f noise in human cognition. , 1995, Science.

[77]  R. Ivry,et al.  Perception and production of temporal intervals across a range of durations: evidence for a common timing mechanism. , 1995, Journal of experimental psychology. Human perception and performance.

[78]  Yoshitaka Nakajima,et al.  Auditory Isochrony: Time Shrinking and Temporal Patterns , 1995, Perception.

[79]  P. Juslin Emotional communication in music viewed through a Brunswikian lens , 1995 .

[80]  Manfred Clynes,et al.  Microstructural musical linguistics: composers' pulses are liked most by the best musicians , 1995, Cognition.

[81]  Anders Friberg Matching the rule parameters of PHRASE ARCH to performances of "Träumerei" : a preliminary study , 1995 .

[82]  Bruno H. Repp,et al.  The Practice of Performance: Musical motion and performance: theoretical and empirical perspectives , 1995 .

[83]  L. Coakley,et al.  The best laid schemes. , 1995, Occupational health; a journal for occupational health nurses.

[84]  N. Todd The kinematics of musical expression , 1995 .

[85]  Johan Sundberg,et al.  TIME DISCRIMINATION IN A MONOTONIC, ISOCHRONOUS SEQUENCE , 1995 .

[86]  Eric Clarke The Practice of Performance: Expression in performance: generativity, perception and semiosis , 1995 .

[87]  Anders Friberg,et al.  A Quantitative Rule System for Musical Performance , 1995 .

[88]  K. Heilman,et al.  Understanding emotional prosody activates right hemisphere regions. , 1996, Archives of neurology.

[89]  John A. Sloboda,et al.  The acquisition of musical performance expertise: Deconstructing the "talent" account of individual differences in musical expressivity. , 1996 .

[90]  Edward T. Bullmore,et al.  A direct demonstration of functional specialization within motion-related visual and auditory cortex of the human brain , 1996, Current Biology.

[91]  J. Vroomen,et al.  Categorical perception of emotional speech , 1996 .

[92]  S. Shamma,et al.  Analysis of dynamic spectra in ferret primary auditory cortex. I. Characteristics of single-unit responses to moving ripple spectra. , 1996, Journal of neurophysiology.

[93]  S. Shamma,et al.  Analysis of dynamic spectra in ferret primary auditory cortex. II. Prediction of unit responses to arbitrary dynamic spectra. , 1996, Journal of neurophysiology.

[94]  P. Juslin,et al.  Emotional Expression in Music Performance: Between the Performer's Intention and the Listener's Experience , 1996 .

[95]  C. Palmer Anatomy of a Performance: Sources of Musical Expression , 1996 .

[96]  P. Juslin Emotional Communication in Music Performance: A Functionalist Perspective and Some Data , 1997 .

[97]  Bruno H. Repp,et al.  Variability of timing in expressive piano performance increases with interval duration , 1997 .

[98]  Patrik N. Juslin,et al.  Can results from studies of perceived expression in musical performances be generalized across response formats , 1997 .

[99]  B H Repp,et al.  Acoustics, perception, and production of legato articulation on a computer-controlled grand piano. , 1997, The Journal of the Acoustical Society of America.

[100]  P. Juslin Perceived Emotional Expression in Synthesized Performances of a Short Melody: Capturing the Listener's Judgment Policy , 1997 .

[101]  Philip J. Benson,et al.  Categorical Perception of Facial Expressions: Categories and their Internal Structure , 1997 .

[102]  B. Repp The Aesthetic Quality of a Quantitatively Average Music Performance: Two Preliminary Experiments , 1997 .

[103]  Carolyn Drake,et al.  Sources of timing variations in music performance: A psychological segmentation model , 1998 .

[104]  Derek Matravers Art and emotion , 1998 .

[105]  Patrik N. Juslin A functionalist perspective on emotional communication in music performance , 1998 .

[106]  Johan Sundberg,et al.  Musical punctuation on the microlevel : Automatic identification and performance of small melodic units , 1998 .

[107]  L. Obler,et al.  Right hemisphere emotional perception: evidence across multiple channels. , 1998, Neuropsychology.

[108]  Luke Windsor,et al.  Computational Modeling of Music Cognition: Problem or Solution? , 1998 .

[109]  I. Peretz,et al.  Music and emotion: perceptual determinants, immediacy, and isolation after brain damage , 1998, Cognition.

[110]  P. Juslin,et al.  The Role of Timing Patterns in Recognition of Emotional Expression from Musical Performance , 1999 .

[111]  J. Sundberg,et al.  Does music performance allude to locomotion? A model of final ritardandi derived from measurements of stopping , 1999 .

[112]  Max Coltheart,et al.  Modularity and cognition , 1999, Trends in Cognitive Sciences.

[113]  Neil P. McAngus Todd,et al.  Motion in Music: A Neurobiological Perspective , 1999 .

[114]  P. Laukka,et al.  Improving Emotional Communication in Music Performance through Cognitive Feedback , 2000 .

[115]  Johan Sundberg,et al.  Generating Musical Performances with Director Musices , 2000, Computer Music Journal.

[116]  Guy Madison On the nature of variability in isochronous serial interval production , 2000 .

[117]  Johan Sundberg,et al.  Music from Motion: Sound Level Envelopes of Tones Expressing Human Locomotion , 2000 .

[118]  W. L. Windsor,et al.  Real and Simulated Expression: A Listening Study , 2000 .

[119]  E. Bullmore,et al.  The functional neuroanatomy of implicit-motion perception or ‘representational momentum’ , 2000, Current Biology.

[120]  T. Allison,et al.  Social perception from visual cues: role of the STS region , 2000, Trends in Cognitive Sciences.

[121]  Anders Friberg,et al.  Emotional Coloring of Computer-Controlled Music Performances , 2000, Computer Music Journal.

[122]  P. Juslin,et al.  Cue Utilization in Communication of Emotion in Music Performance: Relating Performance to Perception Studies of Music Performance , 2022 .

[123]  Johan Sundberg Computer synthesis of music performance , 2001 .

[124]  A. Friederici,et al.  Musical syntax is processed in Broca's area: an MEG study , 2001, Nature Neuroscience.

[125]  D. Gilden Cognitive emissions of 1/f noise. , 2001, Psychological review.

[126]  P. Laukka,et al.  Impact of intended emotion intensity on cue utilization and decoding accuracy in vocal expression of emotion. , 2001, Emotion.

[127]  K. Scherer,et al.  Emotion Inferences from Vocal Expression Correlate Across Languages and Cultures , 2001 .

[128]  I. Peretz Listen to the brain: A biological perspective on musical emotions. , 2001 .

[129]  P. Juslin Communicating emotion in music performance: A review and a theoretical framework , 2001 .

[130]  A. Bregman Auditory Scene Analysis , 2001 .

[131]  J. Sloboda,et al.  Psychological perspectives on music and emotion , 2001 .

[132]  Guy Madison,et al.  Functional Modelling of the Human Timing Mechanism , 2001 .

[133]  R. Zatorre,et al.  Structure and function of auditory cortex: music and speech , 2002, Trends in Cognitive Sciences.

[134]  P. Juslin,et al.  Emotional expression in music. , 2003 .