Bob: an improvisational music companion

In this thesis, I introduce a new melody representation scheme and a machine learning framework that enables customized interaction between a live, improvising musician and the computer. The ultimate intent of these technologies is to provide the infrastructure needed to intimately couple the computer with a musician's all-too-transient improvisations; potential applications range from improvisational exploration to education and musical analysis. I introduce Band-OUT-of-a-Box (BoB) | a fully realized agent that trades personalized solos with a simulated user in realtime. Musical improvisation is an ill-de ned situationand user-speci c practice whose inherent non-literal basis makes the authoring techniques exploited in other AI/entertainment systems (e.g., interactive characters or stories) less helpful when building an improvisational music companion. A major contribution is BoB's computational melodic improvisation model, which con gures itself to its users' unlabeled examples, alleviating some of the burden associated with de ning an appropriate musical aesthetic. I rst describe an abstract perception algorithm that maps short strings of notes onto a mixture model. The components of this model correspond to the various playing modes | e.g., tonal, intervallic, and directional trends | that the user employed during various parts of a warmup session. I next describe a crucial technology that closes the perception loop by integrating the learned model's parameters into a stochastic process that, when sampled, can produce sequences that exhibit speci c abstract goals (or playing modes) while seamlessly integrating into the constraints set up by the local environment. These algorithms' musical performances are evaluated by qualitatively exploring their behavior using two di erent simulations: both the transcriptions of Bebop saxophonist Charlie Parker and jazz violinist Stephane Grappelli. These algorithm's quantitative performance are also assessed using more traditional machine learning techniques.

[1]  Wolfram Menzel,et al.  HARMONET: A Neural Net for Harmonizing Chorales in the Style of J. S. Bach , 1991, NIPS.

[2]  Roger B. Dannenberg,et al.  Combining Instrument and Performance Models for High-Quality Music Synthesis , 1998 .

[3]  R. Malina Aaron’s Code: Meta-Art, Artificial Intelligence and the Work of Harold Cohen by Pamela McCorduck (review) , 2017 .

[4]  Bruce W. Pennycook,et al.  Real-time Recognition of Melodic Fragments Using the Dynamic Timewarp Algorithm , 1993, ICMC.

[5]  John A. Biles,et al.  GenJam Populi: Training an IGA via Audience-Mediated Performance , 1995, ICMC.

[6]  William F. Walker,et al.  A computer participant in musical improvisation , 1997, CHI.

[7]  Dominik Hörnel A multi-scale neural-network model for learnig and reproducing chorale variations , 1998 .

[8]  Padhraic Smyth,et al.  Model Complexity, Goodness of Fit and Diminishing Returns , 2000, NIPS.

[9]  Ali Taylan Cemgil,et al.  Tempo tracking and rhythm quantization by sequential Monte Carlo , 2001, NIPS.

[10]  Andrew Stern,et al.  Socially intelligent virtual petz , 1997 .

[11]  Joseph Bates,et al.  Guiding interactive drama , 1997 .

[12]  Karin Höthker Modelling the Motivic Process of Melodies with Markov Chains , 1999, ICMC.

[13]  Anil K. Jain,et al.  Clustering Methodologies in Exploratory Data Analysis , 1980, Adv. Comput..

[14]  Ran El-Yaniv,et al.  Universal Classification Applied to Musical Sequences , 1998, ICMC.

[15]  M. Degroot Optimal Statistical Decisions , 1970 .

[16]  Luke Windsor,et al.  Make Me a Match: An Evaluation of Different Approaches to ScorePerformance Matching , 2000, Computer Music Journal.

[17]  Aaron Bryan Loyall,et al.  Believable agents: building interactive personalities , 1997 .

[18]  Ali Taylan Cemgil,et al.  Bayesian Real-time Adaptation for Interactive Performance Systems , 2001, ICMC.

[19]  Paul R. Cohen,et al.  Multivariate Clustering by Dynamics , 2000, AAAI/IAAI.

[20]  Ali Taylan Cemgil,et al.  Robust Time-quantization for Music, from Performance to Score , 1999 .

[21]  Joanna J. Bryson,et al.  The Subsumption Strategy Development of a Music Modelling System , 1992 .

[22]  Ali S. Hadi,et al.  Finding Groups in Data: An Introduction to Chster Analysis , 1991 .

[23]  Mark C. Gridley,et al.  Jazz Styles: History and Analysis , 1978 .

[24]  P. Berliner Thinking in Jazz: The Infinite Art of Improvisation , 1995 .

[25]  Shlomo Dubnov,et al.  Automatic Modeling of Musical Style , 2001, ICMC.

[26]  Jean-Gabriel Ganascia,et al.  Musical Pattern Extraction and Similarity Assessment , 2000, Readings in Music and Artificial Intelligence.

[27]  Michael Mateas,et al.  An Oz-Centric Review of Interactive Drama and Believable Agents , 1999, Artificial Intelligence Today.

[28]  Sebastian Thrun,et al.  Learning to Classify Text from Labeled and Unlabeled Documents , 1998, AAAI/IAAI.

[29]  M. Mateas Expressive AI , 2000 .

[30]  Gerhard Widmer,et al.  Large-scale Induction of Expressive Performance Rules: First Quantitative Results , 2000, ICMC.

[31]  Dexter Kozen,et al.  The Design and Analysis of Algorithms , 1991, Texts and Monographs in Computer Science.

[32]  B. Hayes-Roth,et al.  Story-Making with Improvisational Puppets and Actors , 1996 .

[33]  Robert Rowe Interactive Music Systems in Ensemble Performance , 2000, Readings in Music and Artificial Intelligence.

[34]  John A. Biles,et al.  Interactive GenJam: Integrating Real-time Performance with a Genetic Algorithm , 1998, ICMC.

[35]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[36]  Rodney A. Brooks,et al.  Elephants don't play chess , 1990, Robotics Auton. Syst..

[37]  G. Casella,et al.  Statistical Inference , 2003, Encyclopedia of Social Network Analysis and Mining.

[38]  E. Narmour The analysis and cognition of basic melodic structures , 1992 .

[39]  Peter M. Todd,et al.  Connectionism and Musiconomy , 2003 .

[40]  D. N. Geary Mixture Models: Inference and Applications to Clustering , 1989 .

[41]  Phoebe Sengers,et al.  Antiboxology: agent design in cultural context , 1998 .

[42]  David Levitt,et al.  A melody description system for jazz improvisation , 1981 .

[43]  Jaime G. Carbonell,et al.  Interactive drama, art and artificial intelligence , 2002 .

[44]  Padhraic Smyth,et al.  Clustering Sequences with Hidden Markov Models , 1996, NIPS.

[45]  Robert Rowe,et al.  Interactive Music Systems: Machine Listening and Composing , 1992 .

[46]  Belinda Thom,et al.  Artificial Intelligence and Real-Time Interactive Improvisation , 2000 .

[47]  Xavier Serra,et al.  SaxEx: a case-based reasoning system for generating expressive musical performances , 1998, ICMC.

[48]  John A. Biles,et al.  GenJam: A Genetic Algorithm for Generating Jazz Solos , 1994, ICMC.

[49]  David S. Watson,et al.  A Machine Learning Approach to Musical Style Recognition , 1997, ICMC.

[50]  Wolfram Menzel,et al.  Learning musical structure and style with neural networks , 1998 .

[51]  Padhraic Smyth,et al.  A General Probabilistic Framework for Clustering Individuals , 2000, KDD 2000.

[52]  Christopher Raphael,et al.  Music Plus One: A System for Expressive and Flexible Musical Accompaniment , 2001, ICMC.

[53]  Paul R. Cohen,et al.  Sequence Learning via Bayesian Clustering by Dynamics , 2001, Sequence Learning.

[54]  Dominik Hörnel,et al.  Melonet: Neural Networks that Learn Harmony-Based Melodic Variations , 1994, ICMC.

[55]  Matthew Wright,et al.  Preparation for Improvised Performance in Collaboration with a Khyal Singer , 1998, ICMC.

[56]  David Sudnow,et al.  Ways of the hand : the organization of improvised conduct , 1978 .

[57]  Marilyn Gail Boltz The Processing of Melodic and Temporal Information: Independent or Unified Dimensions? , 1999 .

[58]  R. Jackendoff,et al.  A Generative Theory of Tonal Music , 1985 .

[59]  Todd Winkler Composing Interactive Music: Techniques and Ideas Using Max , 1998 .

[60]  Christopher Raphael,et al.  Automatic Segmentation of Acoustic Musical Signals Using Hidden Markov Models , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[61]  Derek Bailey Improvisation: Its nature and practice in music , 1980 .

[62]  Ichiro Fujinaga,et al.  A Comparative Latency Study of Hardware and Software Pitch-trackers , 1999, ICMC.

[63]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[64]  Clark Elliott,et al.  Autonomous Agents as Synthetic Characters , 1998, AI Mag..

[65]  P. Johnson-Laird Jazz Improvization: A Theory at the Computational Level , 1991 .

[66]  Anders Friberg,et al.  Emotional Coloring of Computer-Controlled Music Performances , 2000, Computer Music Journal.

[67]  Shlomo Dubnov,et al.  Guessing the Composer's Mind: Applying Universal Prediction to Musical Style , 1999, ICMC.

[68]  A. S. Weigend,et al.  Results of the time series prediction competition at the Santa Fe Institute , 1993, IEEE International Conference on Neural Networks.

[69]  Marina Meila,et al.  An Experimental Comparison of Several Clustering and Initialization Methods , 1998, UAI.

[70]  Dominik Hörnel,et al.  Statistical vs. Connectionist Models of Bebop Improvisation , 1999, ICMC.

[71]  Geraint A. Wiggins,et al.  AI Methods for Algorithmic Composition: A Survey, a Critical View and Future Prospects , 1999 .

[72]  James L. Wright,et al.  Higher-level Composition Control in Music Sketcher: Modifiers and Smart Harmony , 1999, ICMC.

[73]  Judy A. Franklin Multi-Phase Learning for Jazz Improvisation and Interaction , 2001 .

[74]  Jerry Coker The Jazz Idiom , 1975 .

[75]  Costas S. Iliopoulos,et al.  Pattern Processing in Melodic Sequences: Challenges, Caveats and Prospects , 2001, Comput. Humanit..

[76]  David C. McGuire Understanding Jazz , 1978 .

[77]  Peter Desain,et al.  Rhythm Quantization for Transcription , 2000, Computer Music Journal.

[78]  François Pachet,et al.  Computer Analysis of Jazz Chord Sequence: Is Solar a Blues? , 2000, Readings in Music and Artificial Intelligence.

[79]  Gerhard Widmer,et al.  Recognition and Exploitation of Contextual CLues via Incremental Meta-Learning , 1996, ICML.

[80]  Thomas G. Dietterich What is machine learning? , 2020, Archives of Disease in Childhood.

[81]  Belinda Thom Learning melodic models for interactive melodic improvisation , 1999 .

[82]  David Sankoff,et al.  Comparison of musical sequences , 1990, Comput. Humanit..

[83]  Francesco Giomi,et al.  An Interactive System for Musical Improvisation , 1988, ICMC.

[84]  Robert O. Gjerdingen,et al.  The psychology of music , 2002 .

[85]  David Cope,et al.  Computer Modeling of Musical Intelligence in EMI , 1992 .

[86]  Yishay Mansour,et al.  An Information-Theoretic Analysis of Hard and Soft Assignment Methods for Clustering , 1997, UAI.

[87]  M. Minsky Music, Mind, and Meaning , 1981 .

[88]  L. Rabiner,et al.  An introduction to hidden Markov models , 1986, IEEE ASSP Magazine.

[89]  Petri Toiviainen Modeling the Target-Note Technique of Bebop-Style Jazz Improvisation: An Artificial Neural Network Approach , 1995 .

[90]  Emilios Cambouropoulos,et al.  The Local Boundary Detection Model (LBDM) and its Application in the Study of Expressive Timing , 2001, ICMC.

[91]  Roger B. Dannenberg,et al.  Predicting Chords in Jazz , 1995, ICMC.

[92]  Roger B. Dannenberg,et al.  Following an Improvisation in Real Time , 1987, ICMC.

[93]  J. Bharucha Tonality and expectation. , 1994 .

[94]  Sheldon M. Ross,et al.  Introduction to probability models , 1975 .

[95]  Matthew Wright,et al.  Problems and prospects for intimate musical control of computers , 2001 .

[96]  Jean-Gabriel Ganascia,et al.  Automated Motive-Oriented Analysis of Musical Corpuses: a Jazz Case Study , 1996, ICMC.

[97]  M. Minsky The Society of Mind , 1986 .

[98]  Peter E. Hart,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[99]  C. Krumhansl Cognitive Foundations of Musical Pitch , 1990 .

[100]  F. A. Seiler,et al.  Numerical Recipes in C: The Art of Scientific Computing , 1989 .

[101]  J. Elman Distributed Representations, Simple Recurrent Networks, And Grammatical Structure , 1991 .

[102]  Belinda Thom,et al.  Unsupervised Learning and Interactive Jazz/Blues Improvisation , 2000, AAAI/IAAI.

[103]  G. Russell,et al.  調性組織におけるリディアン・クロマティック・コンセプト = The lydian chromatic concept of tonal organization , 1993 .

[104]  Michael C. Mozer,et al.  Neural Network Music Composition by Prediction: Exploring the Benefits of Psychoacoustic Constraints and Multi-scale Processing , 1994, Connect. Sci..

[105]  Paul R. Cohen,et al.  Unsupervised clustering of robot activities: a Bayesian approach , 2000, AGENTS '00.

[106]  Padhraic Smyth,et al.  Probabilistic Model-Based Clustering of Multivariate and Sequential Data , 1999 .