Computational Intelligence in Music Composition: A Survey

Composing music is an inspired yet challenging task, in that the process involves many considerations such as assigning pitches, determining rhythm, and arranging accompaniment. Algorithmic composition aims to develop algorithms for music composition. Recently, algorithmic composition using artificial intelligence technologies received considerable attention. In particular, computational intelligence is widely used and achieves promising results in the creation of music. This paper attempts to provide a survey on the computational intelligence techniques used in music composition. First, the existing approaches are reviewed in light of the major musical elements considered in composition, to wit, musical form, melody, and accompaniment. Second, the review highlights the components of evolutionary algorithms and neural networks designed for music composition.

[1]  Walter D. Potter,et al.  GA-based Music Arranging for Guitar , 2006, 2006 IEEE International Conference on Evolutionary Computation.

[2]  Riccardo Poli,et al.  Particle swarm optimization , 1995, Swarm Intelligence.

[3]  Brad Johanson,et al.  GP-Music: An Interactive Genetic Programming System for Music Generation with Automated Fitness Raters , 2007 .

[4]  Hubert S. Howe Composing by computer , 1975 .

[5]  I. Xenakis,et al.  Formalized Music: Thought and Mathematics in Composition , 1971 .

[6]  Yaochu Jin,et al.  Surrogate-assisted evolutionary computation: Recent advances and future challenges , 2011, Swarm Evol. Comput..

[7]  Takehisa Onisawa,et al.  Music composition system with human evaluation as human centered system , 2003, Soft Comput..

[8]  Wei Li,et al.  Music content authentication based on beat segmentation and fuzzy classification , 2013, EURASIP J. Audio Speech Music. Process..

[9]  Sanghoon Jun,et al.  A fuzzy inference-based music emotion recognition system , 2008 .

[10]  P. M. Gibson,et al.  NEUROGEN, musical composition using genetic algorithms and cooperating neural networks , 1991 .

[11]  Bruce Jacob,et al.  Composing with Genetic Algorithms , 1995, ICMC.

[12]  Takehisa Onisawa,et al.  Music/lyrics composition system considering user's image and music genre , 2009, 2009 IEEE International Conference on Systems, Man and Cybernetics.

[13]  Kwang-Ho Kim,et al.  Music copyright protection system using fuzzy similarity measure for music phoneme segmentation , 2009, 2009 IEEE International Conference on Fuzzy Systems.

[14]  K. Hsü,et al.  Fractal geometry of music. , 1990, Proceedings of the National Academy of Sciences of the United States of America.

[15]  Yoichiro Maeda,et al.  Automatic generation method of twelve tone row for musical composition used genetic algorithm , 2009, 2009 IEEE International Conference on Fuzzy Systems.

[16]  T. A. Maddox,et al.  Using an Evolutionary Algorithm to Generate Four-Part 18th Century Harmony , 2022 .

[17]  Yaochu Jin,et al.  A comprehensive survey of fitness approximation in evolutionary computation , 2005, Soft Comput..

[18]  Roseli A. Francelin Romero,et al.  Generation of composed musical structures through recurrent neural networks based on chaotic inspiration , 2011, The 2011 International Joint Conference on Neural Networks.

[19]  Parag Chordia,et al.  Modeling Melodic Improvisation in Turkish Folk Music Using Variable-Length Markov Models , 2011, ISMIR.

[20]  Gerhard Nierhaus,et al.  Algorithmic Composition: Paradigms of Automated Music Generation , 2008 .

[21]  Chien-Hung Liu,et al.  A Novel Automatic Composition System Using Evolutionary Algorithm and Phrase Imitation , 2017, IEEE Systems Journal.

[22]  Gustavo Diaz-Jerez,et al.  Composing with Melomics: Delving into the Computational World for Musical Inspiration , 2011, Leonardo Music Journal.

[23]  Zhen Wang,et al.  Emotional Music Generation Using Interactive Genetic Algorithm , 2008, 2008 International Conference on Computer Science and Software Engineering.

[24]  George Papadopoulos,et al.  A Genetic Algorithm for the Generation of Jazz Melodies , 2000 .

[25]  Man-Kwan Shan,et al.  Computer Music Composition Based on Discovered Music Patterns , 2006, 2006 IEEE International Conference on Systems, Man and Cybernetics.

[26]  Chunru Wan,et al.  Content-based audio classification and retrieval using a fuzzy logic system: towards multimedia search engines , 2002, Soft Comput..

[27]  Marco Dorigo,et al.  Ant system: optimization by a colony of cooperating agents , 1996, IEEE Trans. Syst. Man Cybern. Part B.

[28]  Makoto Fukumoto,et al.  Proposal for Automated Creation of Drum's Fill-In Pattern Using Interactive Genetic Algorithm , 2011, 2011 International Conference on Biometrics and Kansei Engineering.

[29]  Charles Ames,et al.  The Markov Process as a Compositional Model: A Survey and Tutorial , 2017 .

[30]  Manuel Cebrián,et al.  A simple genetic algorithm for music generation by means of algorithmic information theory , 2007, 2007 IEEE Congress on Evolutionary Computation.

[31]  Bill Manaris,et al.  Monterey Mirror : Combining Markov Models , Genetic Algorithms , and Power Laws An Experiment in Interactive Evolutionary Music Performance , 2011 .

[32]  Giovanni Acampora,et al.  Memetic Music Composition , 2016, IEEE Transactions on Evolutionary Computation.

[33]  Dragan Matic A GENETIC ALGORITHM FOR COMPOSING MUSIC , 2010 .

[34]  Bernhard Sendhoff,et al.  Generalizing Surrogate-Assisted Evolutionary Computation , 2010, IEEE Transactions on Evolutionary Computation.

[35]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[36]  Michael N. Vrahatis,et al.  Interactive music composition driven by feature evolution , 2016, SpringerPlus.

[37]  Justin Donaldson,et al.  Uncovering Affinity of Artists to Multiple Genres from Social Behaviour Data , 2008, ISMIR.

[38]  Bin Xu,et al.  An emotional harmony generation system , 2010, IEEE Congress on Evolutionary Computation.

[39]  Nao and Iba Hitoshi Tokui,et al.  Music Composition with Interactive Evolutionary Computation , 2000 .

[40]  Eleni E. Mangina,et al.  Dynamic Techniques for Genetic Algorithm–Based Music Systems , 2009, Computer Music Journal.

[41]  A. Schoenberg,et al.  Fundamentals of Musical Composition , 1973 .

[42]  John A. Biles Straight-Ahead Jazz with GenJam: A Quick Demonstration , 2013, MUME@AIIDE.

[43]  Ramón López de Mántaras,et al.  Ai and Music: From Composition to Expressive Performance , 2002, AI Mag..

[44]  Hewijin Christine Jiau,et al.  An improved music representation method by using harmonic-based chord decision algorithm , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[45]  Somnuk Phon-Amnuaisuk,et al.  Evolving Musical Harmonisation , 1999, ICANNGA.

[46]  Mark Steedman,et al.  A Generative Grammar for Jazz Chord Sequences , 1984 .

[47]  K J Hsü,et al.  Self-similarity of the "1/f noise" called music. , 1991, Proceedings of the National Academy of Sciences of the United States of America.

[48]  Shlomo Dubnov,et al.  Using Machine-Learning Methods for Musical Style Modeling , 2003, Computer.

[49]  Tanya Vladimirova,et al.  Genetic Algorithm Utilising Neural Network Fitness Evaluation for Musical Composition , 1997, ICANNGA.

[50]  Alexander F. Gelbukh,et al.  Music Composition Based on Linguistic Approach , 2010, MICAI.

[51]  Peter M. Todd,et al.  Modeling the Perception of Tonal Structure with Neural Nets , 1989 .

[52]  Tomasz Michal Oliwa,et al.  Genetic algorithms and the abc music notation language for rock music composition , 2008, GECCO '08.

[53]  Luca Maria Gambardella,et al.  Ant colony system: a cooperative learning approach to the traveling salesman problem , 1997, IEEE Trans. Evol. Comput..

[54]  Sung-Bae Cho,et al.  A Context-Aware Music Recommendation System Using Fuzzy Bayesian Networks with Utility Theory , 2006, FSKD.

[55]  Ernest A. Edmonds,et al.  Cellular automata in generative electronic music and sonic art: a historical and technical review , 2005, Digit. Creativity.

[56]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[57]  Judy A. Franklin,et al.  Recurrent Neural Networks for Music Computation , 2006, INFORMS J. Comput..

[58]  Eleni E. Mangina,et al.  Jazz Sebastian Bach: A GA System for Music Style Modification , 2006, 2006 International Conference on Systems and Networks Communications (ICSNC'06).

[59]  Martin Dostál,et al.  Genetic Algorithms as a Model of Musical Creativity - on Generating of a Human-Like Rhythmic Accompaniment , 2005, Comput. Artif. Intell..

[60]  E. Miranda,et al.  A survey of computer systems for expressive music performance , 2009, CSUR.

[61]  Tim Blackwell,et al.  Swarming and Music , 2007 .

[62]  Eduardo Miranda,et al.  Evolutionary Computer Music , 2007 .

[63]  Yuko Osana,et al.  Automatic melody generation considering chord progression by genetic algorithm , 2014, 2014 Sixth World Congress on Nature and Biologically Inspired Computing (NaBIC 2014).

[64]  Eduardo Reck Miranda,et al.  Constraint programming systems for modeling music theories and composition , 2011, CSUR.

[65]  Yuko Osana,et al.  Automatic composition system using genetic algorithm and N-gram model - influence of N in N-gram model , 2011, 2011 IEEE International Conference on Systems, Man, and Cybernetics.

[66]  Liang Zhao,et al.  Characterizing chaotic melodies in automatic music composition. , 2010, Chaos.

[67]  Chien-Hung Liu,et al.  A novel genetic algorithm considering measures and phrases for generating melody , 2014, 2014 IEEE Congress on Evolutionary Computation (CEC).

[68]  Liang Zhao,et al.  Computer-aided music composition with LSTM neural network and chaotic inspiration , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[69]  Hideyuki Takagi,et al.  Interactive evolutionary computation: fusion of the capabilities of EC optimization and human evaluation , 2001, Proc. IEEE.

[70]  Chien-Hung Liu,et al.  Evolutionary composition using music theory and charts , 2013, 2013 IEEE Symposium on Computational Intelligence for Creativity and Affective Computing (CICAC).

[71]  I-Ting Liu,et al.  Bach in 2014: Music Composition with Recurrent Neural Network , 2014, ArXiv.

[72]  Marc Leman,et al.  A User-Oriented Approach to Music Information Retrieval , 2006, Content-Based Retrieval.

[73]  Francisco Vilchez Vargas,et al.  Artificial musical pattern generation with genetic algorithms , 2015, 2015 Latin America Congress on Computational Intelligence (LA-CCI).

[74]  Takehisa Onisawa,et al.  Music composition by interaction between human and computer , 2009, New Generation Computing.

[75]  Jürgen Schmidhuber,et al.  Finding temporal structure in music: blues improvisation with LSTM recurrent networks , 2002, Proceedings of the 12th IEEE Workshop on Neural Networks for Signal Processing.

[76]  Joongsu Yoon,et al.  An interactive evolutionary system "IFGAM" , 2004, 30th Annual Conference of IEEE Industrial Electronics Society, 2004. IECON 2004.

[77]  John R. Koza,et al.  Genetic programming - on the programming of computers by means of natural selection , 1993, Complex adaptive systems.

[78]  David S. Watson,et al.  A Machine Learning Approach to Musical Style Recognition , 1997, ICMC.

[79]  Lee Spector,et al.  Induction and Recapitulation of Deep Musical Structure , 1995, IJCAI 1995.

[80]  Penousal Machado,et al.  The Art of Artificial Evolution: A Handbook on Evolutionary Art and Music , 2007 .

[81]  Takehisa Onisawa,et al.  Composition of melody reflecting user's feeling , 2000, 2000 26th Annual Conference of the IEEE Industrial Electronics Society. IECON 2000. 2000 IEEE International Conference on Industrial Electronics, Control and Instrumentation. 21st Century Technologies.

[82]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[83]  Sivaji Bandyopadhyay,et al.  Music Genre Classification: A Semi-supervised Approach , 2013, MCPR.

[84]  Risto Miikkulainen,et al.  Creating melodies with evolving recurrent neural networks , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[85]  Tatsuo Unemi,et al.  A tool for composing short music pieces by means of breeding , 2001, 2001 IEEE International Conference on Systems, Man and Cybernetics. e-Systems and e-Man for Cybernetics in Cyberspace (Cat.No.01CH37236).

[86]  Robin Laney,et al.  Algorithmic music as intelligent game music , 2014 .

[87]  Eduardo Miranda,et al.  Cellular Automata Music: An Interdisciplinary Project , 1993 .

[88]  Yuko Osana,et al.  Automatic composition system using genetic algorithm and N-gram model considering melody blocks , 2012, 2012 IEEE Congress on Evolutionary Computation.

[89]  Tzimeas Dimitrios,et al.  A Dynamic GA-Based Rhythm Generator , 2008 .

[90]  Alexander F. Gelbukh,et al.  Automatic Music Composition with Simple Probabilistic Generative Grammars , 2011, Polibits.

[91]  John W. Sheppard,et al.  Evolving Four-Part Harmony Using Genetic Algorithms , 2011, EvoApplications.

[92]  Michael O'Neill,et al.  Tonality driven piano compositions with grammatical evolution , 2015, 2015 IEEE Congress on Evolutionary Computation (CEC).

[93]  Giovanni Acampora,et al.  A hybrid computational intelligence approach for automatic music composition , 2011, 2011 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE 2011).

[94]  Somnuk Phon-Amnuaisuk,et al.  Evolutionary methods for musical composition , 1998 .

[95]  Michael Edwards,et al.  Algorithmic composition , 2011, Commun. ACM.

[96]  Penousal Machado,et al.  A Corpus-Based Hybrid Approach to Music Analysis and Composition , 2007, AAAI.

[97]  Freitas MELODY HARMONIZATION IN EVOLUTIONARY MUSIC USING MULTIOBJECTIVE GENETIC ALGORITHMS , 2011 .

[98]  Ender Özcan,et al.  A Genetic Algorithm for Generating Improvised Music , 2007, Artificial Evolution.

[99]  Geraint A. Wiggins,et al.  AI Methods for Algorithmic Composition: A Survey, a Critical View and Future Prospects , 1999 .

[100]  A. Calvo,et al.  MELEC: Meta-Level Evolutionary Composer , 2013 .

[101]  Peter G. Anderson,et al.  Neural network fitness functions for a musical IGA , 1996 .

[102]  Xin Wang,et al.  Study on the composition rules for Chinese Jiangnan ditty , 2015, 2015 5th International Conference on Information Science and Technology (ICIST).

[103]  Yi-Hsuan Yang,et al.  Music emotion classification: a fuzzy approach , 2006, MM '06.

[104]  Makoto Fukumoto,et al.  A proposal for optimization of sign sound using interactive differential evolution , 2011, 2011 IEEE International Conference on Systems, Man, and Cybernetics.

[105]  G. G. Stokes "J." , 1890, The New Yale Book of Quotations.

[106]  J. Sundberg,et al.  A Fuzzy Analyzer of Emotional Expression in Music Performance and Body Motion , 2004 .

[107]  R. C. Wong,et al.  T-Music: A melody composer based on frequent pattern mining , 2013, 2013 IEEE 29th International Conference on Data Engineering (ICDE).

[108]  Allen Huang,et al.  Deep Learning for Music , 2016, ArXiv.

[109]  Makoto Fukumoto,et al.  Generation of Appropriate User Chord Development Based on Interactive Genetic Algorithm , 2010, 2010 International Conference on P2P, Parallel, Grid, Cloud and Internet Computing.

[110]  Ryan A. McIntyre,et al.  Bach in a box: the evolution of four part Baroque harmony using the genetic algorithm , 1994, Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence.

[111]  Yoichiro Maeda,et al.  Rhythm generation method for automatic musical composition using genetic algorithm , 2010, International Conference on Fuzzy Systems.

[112]  Chien-Hung Liu,et al.  Music pattern mining for chromosome representation in evolutionary composition , 2015, 2015 IEEE Congress on Evolutionary Computation (CEC).

[113]  John A. Biles,et al.  Life with GenJam: interacting with a musical IGA , 1999, IEEE SMC'99 Conference Proceedings. 1999 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.99CH37028).

[114]  Agostinho C. Rosa,et al.  Music composition using genetic evolutionary algorithms , 2000, Proceedings of the 2000 Congress on Evolutionary Computation. CEC00 (Cat. No.00TH8512).

[115]  Michael C. Mozer,et al.  Neural Network Music Composition by Prediction: Exploring the Benefits of Psychoacoustic Constraints and Multi-scale Processing , 1994, Connect. Sci..

[116]  Xavier Serra,et al.  A Genetic Rule-Based Model of Expressive Performance for Jazz Saxophone , 2008, Computer Music Journal.

[117]  Jose D. Fernández,et al.  AI Methods in Algorithmic Composition: A Comprehensive Survey , 2013, J. Artif. Intell. Res..

[118]  Makoto Fukumoto,et al.  The Efficiency of Interactive Differential Evolution in Creation of Sound Contents: In Comparison with Interactive Genetic Algorithm , 2012, 2012 13th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing.

[119]  Leonard M. Isaacson,et al.  Experimental music: Composition with an electronic computer , 1979 .

[120]  Michael O'Neill,et al.  Elevated Pitch: Automated Grammatical Evolution of Short Compositions , 2009, EvoWorkshops.

[121]  Kemal Ebcioglu,et al.  An Expert System for Harmonizing Four-Part Chorales , 1988, ICMC.

[122]  Peter M. Todd,et al.  A Connectionist Approach To Algorithmic Composition , 1989 .

[123]  Abdollah Homaifar,et al.  Genetic algorithm composes music , 1993, 1993 (25th) Southeastern Symposium on System Theory.

[124]  M. Tomari,et al.  Automatic composition system based on genetic algorithm and N-gram model , 2008, 2008 IEEE International Conference on Systems, Man and Cybernetics.

[125]  Susan Wright,et al.  Towards Melodic Extension Using Genetic Algorithms , 2001, J. Educ. Technol. Soc..

[126]  Ying-Ping Chen,et al.  Evolutionary interactive music composition , 2006, GECCO.

[127]  Martin Middendorf,et al.  An ant colony optimizer for melody creation with baroque harmony , 2007, 2007 IEEE Congress on Evolutionary Computation.

[128]  Chun-Yen Wang,et al.  Semantic content-based music retrieval using audio and fuzzy-music-sense features , 2014, 2014 IEEE International Conference on Granular Computing (GrC).

[129]  R. Voss,et al.  ‘1/fnoise’ in music and speech , 1975, Nature.

[130]  David E. Goldberg,et al.  Genetic Algorithms and Computer-Assisted Music Composition , 1991, ICMC.

[131]  Chuan-Kang Ting,et al.  Polyphonic accompaniment using genetic algorithm with music theory , 2012, 2012 IEEE Congress on Evolutionary Computation.

[132]  John A. Biles,et al.  GenJam: evolution of a jazz improviser , 2001 .

[133]  Ying-ping Chen,et al.  Interactive music composition with the CFE framework , 2007, SEVO.

[134]  J. Sundberg,et al.  Overview of the KTH rule system for musical performance. , 2006 .

[135]  John A. Biles,et al.  GenJam: A Genetic Algorithm for Generating Jazz Solos , 1994, ICMC.

[136]  Bingjun Zhang,et al.  CompositeMap: a novel framework for music similarity measure , 2009, SIGIR.

[137]  C. Roads,et al.  Grammars as Representations for Music , 1979 .

[138]  J. Schmidhuber,et al.  A First Look at Music Composition using LSTM Recurrent Neural Networks , 2002 .

[139]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[140]  A. Sima Uyar,et al.  Automatic music generation using evolutionary algorithms and neural networks , 2011, 2011 International Symposium on Innovations in Intelligent Systems and Applications.

[141]  Andrés Garay Acevedo,et al.  Fugue Composition with Counterpoint Melody Generation Using Genetic Algorithms , 2004, CMMR.

[142]  Peter J. Bentley,et al.  Improvised music with swarms , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[143]  Tatsuo Unemi,et al.  A New Musical Tool for Composition and Play Based on Simulated Breeding , 2001 .

[144]  Takehisa Onisawa,et al.  Composition of music using human evaluation , 2001, 10th IEEE International Conference on Fuzzy Systems. (Cat. No.01CH37297).

[145]  O FranciscoChávezdela,et al.  Fuzzy Rule Based System Ensemble for Music Genre Classification , 2012, EvoMUSART.