Sense in expressive music performance: Data acquisition, computational studies, and models

This chapter gives an introduction into basic strands of current research in expressive music performance. A special focus is given on the various methods to acquire performance data either during a performance (e.g., through computer-monitored instruments) or from audio recordings. We then overview the different computational approaches to formalise and model the various aspects in expressive music performance. Future challenges and open problems are tackled briefly at the end of this chapter.

[1]  J. Sundberg,et al.  Does music performance allude to locomotion? A model of final ritardandi derived from measurements of stopping , 1999 .

[2]  N. Todd A Model of Expressive Timing in Tonal Music , 1985 .

[3]  Gerhard Widmer,et al.  Playing Mozart by Analogy: Learning Multi-level Timing and Dynamics Strategies , 2003 .

[4]  Giovanni De Poli Methodologies for Expressiveness Modelling of and for Music Performance , 2004 .

[5]  P. Laukka,et al.  Expression, Perception, and Induction of Musical Emotions: A Review and a Questionnaire Study of Everyday Listening , 2004 .

[6]  Sylvie Gibet,et al.  Gesture-Based Communication in Human-Computer Interaction , 2001, Lecture Notes in Computer Science.

[7]  B H Repp A microcosm of musical expression. III. Contributions of timing and dynamics to the aesthetic impression of pianists' performances of the initial measures of Chopin's Etude in E major. , 1999, The Journal of the Acoustical Society of America.

[8]  Simon Dixon,et al.  LIVE TRACKING OF MUSICAL PERFORMANCES USING ON-LINE TIME WARPING , 2005 .

[9]  B. Repp Expressive Timing in a Debussy Prelude: A Comparison of Student and Expert Pianists , 1997 .

[10]  B. Repp The Art of Inaccuracy: Why Pianists' Errors Are Difficult to Hear , 1996 .

[11]  Chas. H. Judd Zwei Beitr?ge zur Psychologie des Rhythmus und des Tempo. , 1899 .

[12]  C. P. Heinlein,et al.  Pianoforte Damper-Pedalling under Ten Different Experimental Conditions , 1930 .

[13]  Ramón López de Mántaras,et al.  Ai and Music: From Composition to Expressive Performance , 2002, AI Mag..

[14]  Guerino Mazzola Geometrie der Töne , 1990 .

[15]  Pedro Cano,et al.  Score-Performance Matching Using HMMs , 1999, ICMC.

[16]  Anders Friberg,et al.  Towards a rule-based model for violin vibrato , 2001 .

[17]  B H Repp,et al.  Relational invariance of expressive microstructure across global tempo changes in music performance: An exploratory study , 1994, Psychological research.

[18]  Gerhard Widmer,et al.  Production of staccato articulation in Mozart sonatas played on a grand piano. : Preliminary results , 2000 .

[19]  Isabelle Peretz,et al.  ception and Performance, 26, 1797–1813. Juslin, PN, & Sloboda, JA (2001). Music and emotion: Theory and re-search. New York: Oxford Univer , 2002 .

[20]  Peter Hagmann Das Welte-Mignon-Klavier, die Welte-Philharmonie-Orgel und die Anfänge der Reproduktion von Musik , 1984 .

[21]  C Palmer,et al.  Range of planning in music performance. , 1995, Journal of experimental psychology. Human perception and performance.

[22]  D. Deutsch,et al.  The Psychology of Music , 1983 .

[23]  S. Dixon,et al.  PERCEPTUAL SMOOTHNESS OF TEMPO IN EXPRESSIVELY PERFORMED MUSIC , 2006 .

[24]  Simon Dixon Analysis of Musical Content in Digital Audio , 2004 .

[25]  José Antonio Bowen,et al.  Tempo, duration, and flexibility: Techniques in the analysis of performance , 1996 .

[26]  Xavier Serra,et al.  SaxEx: a case-based reasoning system for generating expressive musical performances , 1998, ICMC.

[27]  J. Sloboda,et al.  Music and emotion: Theory and research , 2001 .

[28]  Thomas P. Rohrer The science and psychology of music performance: creative strategies for teaching and learning , 2004 .

[29]  N. Todd The dynamics of dynamics: A model of musical expression , 1992 .

[30]  John Rink The Practice of Performance: Studies in Musical Interpretation , 1998 .

[31]  Anders Friberg,et al.  Rules for automated performance of ensemble music , 1987 .

[32]  Neil P. McAngus Todd,et al.  A computational model of rubato , 1989 .

[33]  Nicholas Cook Structure and Performance Timing in Bach's C Major Prelude (WTCI): An Empirical Study , 1987 .

[34]  Gerhard Widmer,et al.  Using AI and machine learning to study expressive music performance: project survey and first report , 2001, AI Commun..

[35]  Gerhard Widmer,et al.  EXPLORING EXPRESSIVE PERFORMANCE TRAJECTORIES: SIX FAMOUS PIANISTS PLAY SIX CHOPIN PIECES , 2004 .

[36]  C. Palmer,et al.  Units of knowledge in music performance. , 1993, Journal of experimental psychology. Learning, memory, and cognition.

[37]  Anders Friberg,et al.  Automatic Real-Time Extraction of Musical Expression , 2002, ICMC.

[38]  B. Repp Patterns of note onset asynchronies in expressive piano performance. , 1996, The Journal of the Acoustical Society of America.

[39]  Gerhard Widmer,et al.  Modeling the rational basis of musical expression , 1995 .

[40]  Gerhard Widmer,et al.  Large-scale Induction of Expressive Performance Rules: First Quantitative Results , 2000, ICMC.

[41]  B H Repp,et al.  Acoustics, perception, and production of legato articulation on a computer-controlled grand piano. , 1997, The Journal of the Acoustical Society of America.

[42]  Eric Clarke,et al.  Empirical Methods in the Study of Performance , 2004 .

[43]  Richard Parncutt,et al.  The Science & Psychology of Music Performance , 2002 .

[44]  Marc Leman,et al.  Communicating expressiveness and affect in multimodal interactive systems , 2005, IEEE MultiMedia.

[45]  Eric Clarke,et al.  Some Aspects of Rhythm and Expression in Performances of Erik Satie's "Gnossienne No. 5" , 1985 .

[46]  Roger B. Dannenberg,et al.  An On-Line Algorithm for Real-Time Accompaniment , 1984, ICMC.

[47]  Guerino Mazzola,et al.  Rational composition of performance , 1998 .

[48]  Roberto Bresin,et al.  Articulation Rules For Automatic Music Performance , 2001, ICMC.

[49]  Giovanni De Poli,et al.  EXPRESSIVE CLASSIFIERS AT CSC: AN OVERVIEW OF THE MAIN RESEARCH STREAMS , 2003 .

[50]  Caroline Palmer,et al.  On the Assignment of Structure in Music Performance , 1996 .

[51]  R. Ashley Do[[n't]] Change a Hair for Me: The Art of Jazz Rubato , 2002 .

[52]  T Nakamura,et al.  The communication of dynamics between musicians and listeners through musical performance. , 1989, Perception & psychophysics.

[53]  Caroline Palmer,et al.  Movement amplitude and tempo change in piano performance , 2004 .

[54]  L. H. Shaffer,et al.  Timing in Solo and Duet Piano Performances , 1984 .

[55]  G. L. Collier,et al.  A Study of Timing in Two Louis Armstrong Solos , 2002 .

[56]  Aaron Williamon,et al.  Mastery through imitation: A preliminary study , 2005 .

[57]  B H Repp,et al.  Expressive timing in Schumann's "Träumerei:" an analysis of performances by graduate student pianists. , 1995, The Journal of the Acoustical Society of America.

[58]  Anders Friberg,et al.  A Quantitative Rule System for Musical Performance , 1995 .

[59]  N. Todd The kinematics of musical expression , 1995 .

[60]  B H Repp,et al.  Patterns of expressive timing in performances of a Beethoven minuet by nineteen famous pianists. , 1988, The Journal of the Acoustical Society of America.

[61]  Gerhard Widmer,et al.  Studying a creative act with computers: Music performance studies with automated discovery methods , 2005 .

[62]  Bruno H. Repp On Determining the Basic Tempo of an Expressive Music Performance , 1994 .

[63]  Luke Windsor,et al.  Make Me a Match: An Evaluation of Different Approaches to ScorePerformance Matching , 2000, Computer Music Journal.

[64]  C. P. Heinlein,et al.  The Functional Rôle of Finger Touch and Damper-Pedalling in the Appreciation of Pianoforte Music , 1929 .

[65]  Johan Sundberg,et al.  Threshold and Preference Quantities of Rules for Music Performance , 1991 .

[66]  B H Repp The effect of tempo on pedal timing in piano performance , 1997, Psychological research.

[67]  W. Luke Windsor Data Collection, Experimental Design,and Statistics in Musical Research , 2004 .

[68]  Giovanni De Poli,et al.  Expressiveness detection of music performances in the Kinematics Energy Space , 2004 .

[69]  B H Repp,et al.  The dynamics of expressive piano performance: Schumann's "Träumerei" revisited. , 1996, Journal of the Acoustical Society of America.

[70]  B. Repp A microcosm of musical expression. I. Quantitative analysis of pianists' timing in the initial measures of Chopin's Etude in E major. , 1998, The Journal of the Acoustical Society of America.

[71]  Patrik N. Juslin Studies of music performance: A theoretical analysis of empirical findings , 2003 .

[72]  Alfred Binet,et al.  Recherches graphiques sur la musique , 1895 .

[73]  Eric Clarke,et al.  Timing in the performance of Erik Satie's ‘Vexations’ ☆ , 1982 .

[74]  Anders Friberg,et al.  Swing Ratios and Ensemble Timing in Jazz Performance: Evidence for a Common Rhythmic Pattern , 2002 .

[75]  Daniel P. W. Ellis,et al.  Ground-truth transcriptions of real music from force-aligned MIDI syntheses , 2003, ISMIR.

[76]  David S. Watson,et al.  A Machine Learning Approach to Musical Style Recognition , 1997, ICMC.

[77]  B H Repp,et al.  Acoustics, perception, and production of legato articulation on a digital piano. , 1995, The Journal of the Acoustical Society of America.

[78]  Michael T Turvey,et al.  An early oscillator model: studies on the biodynamics of the piano strike (Bernstein & Popova, 1930). , 2003, Motor control.

[79]  Christopher Raphael,et al.  A Hybrid Graphical Model for Aligning Polyphonic Audio with Musical Scores , 2004, ISMIR.

[80]  Sofia Dahl,et al.  Playing the accent : comparing striking velocity and timing in an ostinato rhythm performed by four drummers , 2004 .

[81]  Gerhard Widmer,et al.  Learning expressive performance: The structure‐level approach , 1996 .

[82]  Roberto Bresin,et al.  Articulation Strategies in Expressive Piano Performance Analysis of Legato, Staccato, and Repeated Notes in Performances of the Andante Movement of Mozart’s Sonata in G Major (K 545) , 2000 .

[83]  A. Gabrielsson Music Performance Research at the Millennium , 2003 .

[84]  Dorothy Swainson,et al.  The Physiological Mechanics of Piano Technique , 1962 .

[85]  Anders Friberg,et al.  Emotional Coloring of Computer-Controlled Music Performances , 2000, Computer Music Journal.

[86]  C. P. Heinlein,et al.  A Discussion of the Nature of Pianoforte Damper-Pedalling Together with an Experimental Study of Some Individual Differences in Pedal Performance , 1929 .

[87]  Peter Desain,et al.  Does expressive timing in music performance scale proportionally with tempo? , 1994 .

[88]  Gerhard Widmer,et al.  In Search of the Horowitz Factor: Interim Report on a Musical Discovery Project , 2002, ALT.

[89]  Johan Sundberg,et al.  Musical Performance: A Synthesis-by-Rule Approach , 1983 .

[90]  Marc Leman,et al.  Correlation of Gestural Musical Audio Cues and Perceived Expressive Qualities , 2003, Gesture Workshop.

[91]  Nico Schuler Freedom and Constraints in Timing and Ornamentation (review) , 2004 .

[92]  Johan Sundberg,et al.  Musical punctuation on the microlevel : Automatic identification and performance of small melodic units , 1998 .

[93]  C. Palmer Mapping musical thought to musical performance. , 1989, Journal of experimental psychology. Human perception and performance.

[94]  Gerhard Widmer,et al.  MATCH: A Music Alignment Tool Chest , 2005, ISMIR.

[95]  Carl E. Seashore,et al.  Psychology of the vibrato in voice and instrument , 1936 .

[96]  B. Repp Diversity and commonality in music performance: an analysis of timing microstructure in Schumann's "Träumerei". , 1992, The Journal of the Acoustical Society of America.

[97]  B H Repp,et al.  Some empirical observations on sound level properties of recorded piano tones. , 1993, The Journal of the Acoustical Society of America.

[98]  Gerhard Widmer,et al.  Discovering simple rules in complex data: A meta-learning algorithm and some surprising musical discoveries , 2003, Artif. Intell..

[99]  Nicholas Cook,et al.  Computational and Comparative Musicology , 2004 .

[100]  Luke Windsor,et al.  The Timing of Grace Notes in Skilled Musical Performance at Different Tempi: A Preliminary Case Study , 2001 .

[101]  Johan Sundberg Four Years of Research on Music and Motion , 2000 .

[102]  Otto Ortmann The Physical Basis of Piano Touch and Tone: An Experimental Investigation of the Effect of the Player's Touch upon the Tone of the Piano , 1925 .

[103]  Nell P. McAngusTodd,et al.  The dynamics of dynamics: A model of musical expression , 1992 .

[104]  Peter Desain,et al.  Timing of Ornaments in the Theme from Beethoven's Paisiello Variations: Empirical Data and a Model , 2002 .

[105]  L. H. Shaffer,et al.  Performances of Chopin, Bach, and Bartok: Studies in motor programming , 1981, Cognitive Psychology.

[106]  Haruhiro Katayose,et al.  Rencon 2004: Turing Test for Musical Expression , 2004, NIME.

[107]  Bruno H. Repp,et al.  Pedal Timing and Tempo in Expressive Piano Performance: A Preliminary Investigation , 1996 .

[108]  Q. D. Bowers,et al.  Encyclopedia of automatic musical instruments , 1972 .

[109]  B. Repp Patterns of expressive timing in performances of a Beethoven minuet by nineteen famous pianists. , 1990 .

[110]  L. H. Shaffer,et al.  Metre and rhythm in piano playing , 1985, Cognition.

[111]  John Rink,et al.  Musical Performance: A Guide to Understanding , 2002 .

[112]  B. Repp The Aesthetic Quality of a Quantitatively Average Music Performance: Two Preliminary Experiments , 1997 .

[113]  Sundberg,et al.  Music and Hearing Quarterly Progress and Status Report Is the musical retard an allusion to physical motion ? , 2007 .

[114]  Gerhard Widmer,et al.  Computational Models of Expressive Music Performance: The State of the Art , 2004 .

[115]  K. Hevner,et al.  The affective value of pitch and tempo in music. , 1937 .

[116]  Hiroshi Kato Automatic musical instrument , 1979 .

[117]  Ju-Lee Hong,et al.  Investigating Expressive Timing and Dynamics in Recorded Cello Performances , 2003 .

[118]  Peter Desain,et al.  Modeling Continuous Aspects of Music Performance: Vibrato and Portamento , 1996 .

[119]  Roberto Bresin,et al.  Measurement and reproduction accuracy of computer-controlled grand pianos. , 2003, The Journal of the Acoustical Society of America.

[120]  C. Palmer Music performance. , 1997, Annual review of psychology.

[121]  W Goebl,et al.  Melody lead in piano performance: expressive device or artifact? , 2001, The Journal of the Acoustical Society of America.

[122]  John Rink In Respect of Performance: The View from Musicology , 2003 .

[123]  Jonas Beskow,et al.  Wavesurfer - an open source speech tool , 2000, INTERSPEECH.

[124]  P. Juslin,et al.  Cue Utilization in Communication of Emotion in Music Performance: Relating Performance to Perception Studies of Music Performance , 2022 .

[125]  David Miles Huber,et al.  The MIDI Manual , 1991 .

[126]  Guerino Mazzola,et al.  Tempo Curves Revisited: Hierarchies of Performance Fields , 1994 .

[127]  Johan Sundberg,et al.  Generating Musical Performances with Director Musices , 2000, Computer Music Journal.

[128]  Anssi Klapuri,et al.  Automatic Music Transcription as We Know it Today , 2004 .

[129]  Simon Dixon Extraction of Musical Performance Parameters from Audio Data , 2000 .

[130]  Simon Dixon,et al.  An On-Line Time Warping Algorithm for Tracking Musical Performances , 2005, IJCAI.

[131]  Eric D. Scheirer,et al.  Using musical knowledge to extract expressive performance information from audio recordings , 1998 .

[132]  Dirk-Jan Povel,et al.  Temporal structure of performed music: Some preliminary observations☆ , 1977 .

[133]  Anders Friberg A fuzzy analyzer of emotional expression in music performance and body motion , 2005 .

[134]  Guerino Mazzola,et al.  Performance and Interpretation , 2002 .

[135]  B H Repp,et al.  A microcosm of musical expression: II. Quantitative analysis of pianists' dynamics in the initial measures of Chopin's Etude in E major. , 1999, The Journal of the Acoustical Society of America.

[136]  M. F.,et al.  Bibliography , 1985, Experimental Gerontology.

[137]  Lie Lu,et al.  Automatic mood detection from acoustic music data , 2003, ISMIR.

[138]  Caroline Palmer The role of interpretive preferences in music performance. , 1992 .

[139]  J. Burt Miner A contribution to the psychology of rhythm. , .

[140]  L. Henry Shaffer How to interpret music. , 1992 .

[141]  Georg Rainer Hofmann,et al.  Geometrie der Töne : Elemente der mathematischen musiktheorie , 1990 .

[142]  C. Palmer,et al.  Investigations in the amplitude of sounded piano tones. , 1991, The Journal of the Acoustical Society of America.

[143]  Simon Dixon,et al.  An open source tool for semi-automatic rhythmic annotation , 2004 .

[144]  Shlomo Dubnov,et al.  The topos of music: geometric logic of concepts, theory, and performance , 2005 .

[145]  Thomas L. Rhea,et al.  Evolution of the Keyboard Interface: The Bosendorfer 290 SE Recording Piano and the Moog Multiply-Touch-Sensitive Keyboards , 1990 .

[146]  A. Gabrielsson,et al.  The influence of musical structure on emotional expression. , 2001 .

[147]  P. Juslin,et al.  The Role of Timing Patterns in Recognition of Emotional Expression from Musical Performance , 1999 .

[148]  Robert Rowe,et al.  Action and Perception in Rhythm and Music , 1989 .

[149]  Gerhard Widmer,et al.  Machine Discoveries: A Few Simple, Robust Local Expression Principles , 2002 .

[150]  J. Davidson Visual Perception of Performance Manner in the Movements of Solo Musicians , 1993 .

[151]  A. Gabrielsson,et al.  Performance of musical rhythm in 3/4 and 6/8 meter , 1983 .

[152]  Yoram Singer,et al.  Learning to Align Polyphonic Music , 2004, ISMIR.

[153]  Anders Friberg Generative Rules for Music Performance : A Formal Description of a Rule System , 1991 .

[154]  J. Sundberg,et al.  A Fuzzy Analyzer of Emotional Expression in Music Performance and Body Motion , 2004 .

[155]  Stephen McAdams,et al.  Analyzing Musical Sound , 2004 .

[156]  C. Palmer Anatomy of a Performance: Sources of Musical Expression , 1996 .

[157]  Carlo Drioli,et al.  Modeling and control of expressiveness in music performance , 2004, Proceedings of the IEEE.

[158]  L. H. Shaffer,et al.  Performances of Chopin, Bach, and Bartok: Studies in motor programming , 1984, Cognitive Psychology.

[159]  John Yen,et al.  FLAME—Fuzzy Logic Adaptive Model of Emotions , 2000, Autonomous Agents and Multi-Agent Systems.

[160]  Johan Sundberg,et al.  Music from Motion: Sound Level Envelopes of Tones Expressing Human Locomotion , 2000 .

[161]  KlangArt-Kongress,et al.  Global village, global brain, global music : KlangArt-Kongress 1999 , 2003 .

[162]  C Palmer,et al.  Harmonic, melodic, and frequency height influences in the perception of multivoiced music , 1994, Perception & psychophysics.

[163]  Luke Windsor,et al.  Data processing in music performance research: Using structural information to improve score-performance matching , 2000, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[164]  Roberto Bresin,et al.  Are computer-controlled pianos a reliable tool in music performance research? Recording and reproduction precision of a Yamaha Disklavier grand piano , 2001 .

[165]  Simon Dixon,et al.  Automatic Extraction of Tempo and Beat From Expressive Performances , 2001 .

[166]  Xavier Rodet,et al.  Improving polyphonic and poly-instrumental music to score alignment , 2003, ISMIR.

[167]  Gerhard Widmer,et al.  In Search of the Horowitz Factor , 2003, AI Mag..

[168]  Gerhard Widmer,et al.  Relational IBL in classical music , 2006, Machine Learning.

[169]  Anders Friberg,et al.  Affect, Attitude and Evaluation of Multisensory Performances , 2005 .

[170]  Hermann Gottschewski Die Interpretation als Kunstwerk : musikalische Zeitgestaltung und ihre Analyse am Beispiel von Welte-Mignon-Klavieraufnahmen aus dem Jahre 1905 , 1996 .

[171]  B H Repp,et al.  Detectability of duration and intensity increments in melody tones: A partial connection between music perception and performance , 1995, Perception & psychophysics.

[172]  Nicola Orio,et al.  Score Following Using Spectral Analysis and Hidden Markov Models , 2001, ICMC.

[173]  W. L. Windsor,et al.  Real and Simulated Expression: A Listening Study , 2000 .