A Survey on Visualizations for Musical Data

Digital methods are increasingly applied to store, structure and analyse vast amounts of musical data. In this context, visualization plays a crucial role, as it assists musicologists and non‐expert users in data analysis and in gaining new knowledge. This survey focuses on this unique link between musicology and visualization. We classify 129 related works according to the visualized data types, and we analyse which visualization techniques were applied for certain research inquiries and to fulfill specific tasks. Next to scientific references, we take commercial music software and public websites into account, that contribute novel concepts of visualizing musicological data. We encounter different aspects of uncertainty as major problems when dealing with musicological data and show how occurring inconsistencies are processed and visually communicated. Drawing from our overview in the field, we identify open challenges for research on the interface of musicology and visualization to be tackled in the future.

[1]  Masataka Goto,et al.  Musicream: New Music Playback Interface for Streaming, Sticking, Sorting, and Recalling Musical Pieces , 2005, ISMIR.

[2]  Junjie Yao,et al.  Single-impulse Panoramic Photoacoustic Computed Tomography of Small-animal Whole-body Dynamics at High Spatiotemporal Resolution , 2017, Nature Biomedical Engineering.

[3]  Peter Knees,et al.  An innovative three-dimensional user interface for exploring music collections enriched , 2006, MM '06.

[4]  Stefan Jänicke,et al.  Untangling the Social Network of Musicians , 2017, DH.

[5]  Rodrigo Fernandes de Mello,et al.  Similarity graph: visual exploration of song collections , 2015 .

[6]  Andreas Butz,et al.  Pulling strings from a tangle: visualizing a personal music listening history , 2009, IUI.

[7]  Stefan Jänicke,et al.  On the Distant Reading of Musicians' Biographies , 2016, DH.

[8]  Peter Knees,et al.  Discovering and Visualizing Prototypical Artists by Web-Based Co-Occurrence Analysis , 2005, ISMIR.

[9]  Elias Pampalk,et al.  Content-based organization and visualization of music archives , 2002, MULTIMEDIA '02.

[10]  Bruce E. Massis “Serendipitous” browsing versus library space , 2011 .

[11]  Jonathan C. Roberts,et al.  The Explanatory Visualization Framework: An Active Learning Framework for Teaching Creative Computing Using Explanatory Visualizations , 2018, IEEE Transactions on Visualization and Computer Graphics.

[12]  Simon Dixon An Interactive Beat Tracking and Visualisation System , 2001, ICMC.

[13]  Mario Nöcker,et al.  Databionic Visualization of Music Collections According to Perceptual Distance , 2005, ISMIR.

[14]  Meinard Müller,et al.  Fundamentals of Music Processing , 2015, Springer International Publishing.

[15]  Yang Liu,et al.  Chorlody: a music learning game , 2014, CHI Extended Abstracts.

[16]  Erik V. Jansson,et al.  From touch to string vibrations. II: The motion of the key and hammer , 1991 .

[17]  M. Mourtzakis,et al.  Assessment of skeletal muscle mass in critically ill patients: considerations for the utility of computed tomography imaging and ultrasonography , 2016, Current opinion in clinical nutrition and metabolic care.

[18]  Ben Shneiderman,et al.  The eyes have it: a task by data type taxonomy for information visualizations , 1996, Proceedings 1996 IEEE Symposium on Visual Languages.

[19]  Sriram Subramanian,et al.  Rouages: Revealing the Mechanisms of Digital Musical Instruments to the Audience , 2013, NIME.

[20]  Greta Franzini,et al.  Visual Text Analysis in Digital Humanities , 2017, Comput. Graph. Forum.

[21]  Stefan Jänicke,et al.  A Timeline Metaphor for Analyzing the Relationships between Musical Instruments and Musical Pieces , 2020, VISIGRAPP.

[22]  Heinrich Schenker,et al.  Five graphic music analyses (Fünf Urlinie-Tafeln) , 1969 .

[23]  Tamara Munzner,et al.  A Multi-Level Typology of Abstract Visualization Tasks , 2013, IEEE Transactions on Visualization and Computer Graphics.

[24]  Stephen W. Smoliar,et al.  pianoFORTE: a system for piano education beyond notation literacy , 1995, MULTIMEDIA '95.

[25]  Yongmeng Wu,et al.  Open Symphony: Creative Participation for Audiences of Live Music Performances , 2017, IEEE Multim..

[26]  Melanie Tory,et al.  Music selection using the PartyVote democratic jukebox , 2008, AVI '08.

[27]  Andreas Butz,et al.  Musicsim: integrating audio analysis and user feedback in an interactive music browsing ui , 2009, IUI.

[28]  Vassilis Kostakos Is the crowd's wisdom biased? A quantitative asessment of three online communities , 2009, ArXiv.

[29]  Danah Boyd,et al.  Vizster: visualizing online social networks , 2005, IEEE Symposium on Information Visualization, 2005. INFOVIS 2005..

[30]  Issei Fujishiro,et al.  comp-i: A System for Visual Exploration and Editing of MIDI Datasets , 2004, ICMC.

[31]  Vassilis Kostakos,et al.  Is the Crowd's Wisdom Biased? A Quantitative Analysis of Three Online Communities , 2009, 2009 International Conference on Computational Science and Engineering.

[32]  Markus Schedl,et al.  A personality-based adaptive system for visualizing classical music performances , 2016, MMSys.

[33]  Masataka Goto,et al.  A chorus section detection method for musical audio signals and its application to a music listening station , 2006, IEEE Transactions on Audio, Speech, and Language Processing.

[34]  Gerik Scheuermann,et al.  Network Visualization in the Humanities (Dagstuhl Seminar 18482) , 2018, Dagstuhl Reports.

[35]  Charl P. Botha,et al.  Piece wise Laplacian‐based Projection for Interactive Data Exploration and Organization , 2011, Comput. Graph. Forum.

[36]  Martin Wattenberg,et al.  Stacked Graphs – Geometry & Aesthetics , 2008, IEEE Transactions on Visualization and Computer Graphics.

[37]  Golan Levin,et al.  In-situ speech visualization in real-time interactive installation and performance , 2004, NPAR '04.

[38]  François Pachet,et al.  Music Similarity Measures: What's the use? , 2002, ISMIR.

[39]  Basheer Tome,et al.  Andante: Walking Figures on the Piano Keyboard to Visualize Musical Motion , 2014, NIME.

[40]  Mark Notess,et al.  Variations 2 : Toward Visual Interfaces for Digital Music Libraries , 2002 .

[41]  David F. Gleich,et al.  The World of Music: SDP layout of high dimensional data , 2005 .

[42]  Menno-Jan Kraak,et al.  Introduction Exploring Geovisualization , 2005 .

[43]  Jonathan Foote,et al.  Visualizing music and audio using self-similarity , 1999, MULTIMEDIA '99.

[44]  Sylvie Ranwez,et al.  Concept Maps for Designing Adaptive Knowledge Maps , 2006, Inf. Vis..

[45]  Daniel Heartz The Répertoire International des Sources Musicales , 1961 .

[46]  Robert Schmitt,et al.  Industrial applications of computed tomography , 2014 .

[47]  Daniel A. Keim,et al.  Analyzing Visual Mappings of Traditional and Alternative Music Notation , 2018, ArXiv.

[48]  Fabio Vignoli,et al.  Visual Playlist Generation on the Artist Map , 2005, ISMIR.

[49]  Dong Liu,et al.  Visual Analyses of Music History: A User-Centric Approach , 2017, ArXiv.

[50]  Stefan Jänicke,et al.  An Interactive Chart of Biography , 2019, 2019 IEEE Pacific Visualization Symposium (PacificVis).

[51]  Ge Wang,et al.  InterFACE: new faces for musical expression , 2018, NIME.

[52]  Carolyn Doi Connecting Music and Place: Exploring Library Collection Data Using Geo-visualizations , 2017 .

[53]  Paul Lamere,et al.  Using 3D Visualizations to Explore and Discover Music , 2007, ISMIR.

[54]  Igor Leão dos Santos,et al.  Semantic Clustering in Wireless Sensor Networks , 2010, WCITD/NF.

[55]  Mark Gahegan,et al.  Visual Semiotics & Uncertainty Visualization: An Empirical Study , 2012, IEEE Transactions on Visualization and Computer Graphics.

[56]  Jiang Hsieh,et al.  Computed Tomography: Principles, Design, Artifacts, and Recent Advances, Fourth Edition , 2022 .

[57]  Jörn Loviscach,et al.  Music Icons: Procedural Glyphs for Audio Files , 2006, 2006 19th Brazilian Symposium on Computer Graphics and Image Processing.

[58]  George Tzanetakis,et al.  ENHANCING SONIC BROWSING USING AUDIO INFORMATION RETRIEVAL , 2002 .

[59]  Nicolas Anton,et al.  Inorganic Nanoparticles Based Contrast Agents for X‐ray Computed Tomography , 2012, Advanced healthcare materials.

[60]  M. Wattenberg The Shape of Song , 2001 .

[61]  L. Hall The language of music , 2006 .

[62]  Marcelo M. Wanderley,et al.  Enhancing the Visualization of Percussion Gestures by Virtual Character Animation , 2008, NIME.

[63]  Andrew H. Bullen Bringing Sheet Music to Life: My Experiences with OMR , 2008 .

[64]  Bert Masschaele,et al.  Nondestructive research on wooden musical instruments : from macro- to microscale imaging with lab-based X-ray CT systems , 2017 .

[65]  Masataka Goto,et al.  Drumix: An Audio Player with Real-time Drum-part Rearrangement Functions for Active Music Listening , 2007 .

[66]  A. Bezerianos,et al.  EduClust - A Visualization Application for Teaching Clustering Algorithms , 2019, Eurographics.

[67]  Ben Shneiderman,et al.  Readings in information visualization - using vision to think , 1999 .

[68]  J. Stephen Downie,et al.  Visual Collaging Of Music In A Digital Library , 2004, ISMIR.

[69]  Gerhard Widmer,et al.  Real Time Tracking and Visualisation of Musical Expression , 2002, ICMAI.

[70]  Sebastian Boring,et al.  The Streams of Our Lives: Visualizing Listening Histories in Context , 2010, IEEE Transactions on Visualization and Computer Graphics.

[71]  Eric J. Isaacson Content Visualization in a Digital Music Library , 2003 .

[72]  Robert S. Laramee,et al.  Survey of Surveys (SoS) ‐ Mapping The Landscape of Survey Papers in Information Visualization , 2017, Comput. Graph. Forum.

[73]  Bozena Kostek,et al.  Music Mood Visualization Using Self-Organizing Maps , 2015 .

[74]  Fabio Vignoli,et al.  Mapping Music In The Palm Of Your Hand, Explore And Discover Your Collection , 2004, ISMIR.

[75]  Huamin Qu,et al.  Visualizing the Semantic Structure in Classical Music Works , 2010, IEEE Transactions on Visualization and Computer Graphics.

[76]  Niko Plath,et al.  3D Imaging of Musical Instruments: Methods and Applications , 2019, Current Research in Systematic Musicology.

[77]  Winnie Wing-Yi Chan A Report on Musical Structure Visualization , 2007 .

[78]  Geoff Wyvill,et al.  Visualization of musical pitch , 2003, Proceedings Computer Graphics International 2003.

[79]  Karrie Karahalios,et al.  Isochords: visualizing structure in music , 2007, GI '07.

[80]  Arzu Çöltekin,et al.  Persistent challenges in geovisualization – a community perspective , 2017 .

[81]  Georg Essl,et al.  Visual Associations in Augmented Keyboard Performance , 2013, NIME.

[82]  Yun-Long Lay,et al.  Development and evaluation of computer-aided music-learning system for the hearing impaired , 2007, J. Comput. Assist. Learn..

[83]  Tsutomu Terada,et al.  A Piano Learning Support System considering rhythm , 2012, ICMC.

[84]  Stefan Jänicke,et al.  Timages : Enhancing Time Graphs with Iconographic Information , 2018 .

[85]  Michael Good MusicXML: An internet-friendly format for sheet music , 2001 .

[86]  A. Halpern Musical expertise and melodic structure in memory for musical notation. , 1982 .

[87]  Golan Levin,et al.  Sounds from Shapes: Audiovisual Performance with Hand Silhouette Contours in The Manual Input Sessions , 2005, NIME.

[88]  Allison Woodruff,et al.  Guidelines for using multiple views in information visualization , 2000, AVI '00.

[89]  Zhouyu Fu,et al.  A Survey of Audio-Based Music Classification and Annotation , 2011, IEEE Transactions on Multimedia.

[90]  Ehud Sharlin,et al.  Vuzik: Music Visualization and Creation on an Interactive Surface , 2011 .

[91]  Deborah I. Fels,et al.  Creating access to music through visualization , 2009, 2009 IEEE Toronto International Conference Science and Technology for Humanity (TIC-STH).

[92]  Justin Donaldson Music Recommendation Mapping and Interface Based on Structural Network Entropy , 2007, 2007 IEEE 23rd International Conference on Data Engineering Workshop.

[93]  Emilia Gómez,et al.  The Musical Avatar: a visualization of musical preferences by means of audio content description , 2010, Audio Mostly Conference.

[94]  Cumhur Erkut,et al.  Virtual Reality Musical Instruments: State of the Art, Design Principles, and Future Directions , 2016, Computer Music Journal.

[95]  Marti A. Hearst,et al.  ImproViz: visual explorations of jazz improvisations , 2005, CHI EA '05.

[96]  Andreas Rauber,et al.  The Map of Mozart , 2006, ISMIR.

[97]  Dominik Lübbers SoniXplorer: Combining Visualization and Auralization for Content-Based Exploration of Music Collections , 2005, ISMIR.

[98]  Silvia Miksch,et al.  Visualization of Cultural Heritage Collection Data: State of the Art and Future Challenges , 2019, IEEE Transactions on Visualization and Computer Graphics.

[99]  Review of the Uses of Computed Tomography for Analyzing Instruments of the Violin Family with a Focus on the Future , 2010 .

[100]  Heidrun Schumann,et al.  A Survey of Multi-faceted Graph Visualization , 2015, EuroVis.

[101]  Gerhard Widmer,et al.  Visualizing changes in the structure of data for exploratory feature selection , 2003, KDD '03.

[102]  T. Huisman,et al.  Computed Tomographic Findings in Microcephaly Associated with Zika Virus. , 2016, The New England journal of medicine.

[103]  Masataka Goto,et al.  MusicRainbow: A New User Interface to Discover Artists Using Audio-based Similarity and Web-based Labeling , 2006, ISMIR.

[104]  Dirk Balfanz,et al.  Mobile liquid 2D scatter space (ML2DSS) , 2004, Proceedings. Eighth International Conference on Information Visualisation, 2004. IV 2004..

[105]  Michelle Urberg Pasts and Futures of Digital Humanities in Musicology: Moving Towards a “Bigger Tent” , 2017 .

[106]  Martin Wattenberg,et al.  Arc diagrams: visualizing structure in strings , 2002, IEEE Symposium on Information Visualization, 2002. INFOVIS 2002..

[107]  Shiliang Zhang,et al.  Affective Visualization and Retrieval for Music Video , 2010, IEEE Transactions on Multimedia.

[108]  Craig Stuart Sapp Harmonic Visualizations of Tonal Music , 2001, ICMC.

[109]  Werner Goebl,et al.  Visualizing Expressive Performance in Tempo—Loudness Space , 2003, Computer Music Journal.

[110]  George Tzanetakis,et al.  MARSYAS3D: A PROTOTYPE AUDIO BROWSER-EDITOR USING A LARGE SCALE IMMERSIVE VISUAL AND AUDIO DISPLAY , 2001 .

[111]  Gerhard Widmer,et al.  Classical Music on the Web - User Interfaces and Data Representations , 2015, ISMIR.

[112]  E. Brazil,et al.  Audio information browsing with the Sonic Browser , 2003, Proceedings International Conference on Coordinated and Multiple Views in Exploratory Visualization - CMV 2003 -.

[113]  Shyh-Kang Jeng,et al.  Emotion-Based Music Visualization Using Photos , 2008, MMM.

[114]  Andreas Kerren,et al.  Text Visualization Browser : A Visual Survey of Text Visualization Techniques , 2014 .

[115]  Pedro Cano,et al.  On the use of FastMap for Audio Retrieval and Browsing , 2002, ISMIR.

[116]  Andreas Butz,et al.  AudioRadar: A Metaphorical Visualization for the Navigation of Large Music Collections , 2006, Smart Graphics.

[117]  Tamara Munzner,et al.  A Nested Model for Visualization Design and Validation , 2009, IEEE Transactions on Visualization and Computer Graphics.

[118]  Masataka Goto,et al.  A Virtual Dancer ‘ ‘ Cindy ’ ’ Interactive Performance of a Music-controlled CG Dancer , 2008 .

[119]  MunznerTamara A Nested Model for Visualization Design and Validation , 2009 .

[120]  Adam J. Sporka,et al.  CoBoard flowers: An interactive installation for Prague spring international music festival , 2016, 2016 7th IEEE International Conference on Cognitive Infocommunications (CogInfoCom).

[121]  Josep Lluís Arcos,et al.  Visualizing and Exploring Personal Music Libraries , 2004, ISMIR.

[122]  Delfina Malandrino,et al.  A Color-Based Visualization Approach to Understand Harmonic Structures of Musical Compositions , 2015, 2015 19th International Conference on Information Visualisation.

[123]  Stefan Leitich,et al.  Globe of Music - Music Library Visualization Using Geosom , 2007, ISMIR.

[124]  Herman Chernoff,et al.  The Use of Faces to Represent Points in k- Dimensional Space Graphically , 1973 .

[125]  Sebastian Kirsch,et al.  Computed Tomography as a Tool for Archiving Ethnomusicological Objects , 2019, Current Research in Systematic Musicology.

[126]  Ben J. Miller Music Learning through Video Games and Apps: Guitar Hero, Rock Band, Amplitude, Frequency, and Rocksmith, and Bandfuse , and Bit.Trip Complete, and Audiosurf, and Beat Hazard, and Biophilia (review) , 2013 .

[127]  Denis Parra,et al.  Moodplay: Interactive Mood-based Music Discovery and Recommendation , 2016, UMAP.

[128]  Monica M. C. Schraefel,et al.  Continuum: designing timelines for hierarchies, relationships and scale , 2007, UIST.

[129]  Marc Leman,et al.  Content-Based Music Information Retrieval: Current Directions and Future Challenges , 2008, Proceedings of the IEEE.

[130]  Peter Knees,et al.  Assigning and Visualizing Music Genres by Web-based Co-Occurrence Analysis , 2006, ISMIR.

[131]  Issei Fujishiro,et al.  BRASS: Visualizing Scores for Assisting Music Learning , 2003, ICMC.

[132]  Maria Cristina Pattuelli,et al.  Visualizing linked Jazz: A web-based tool for social network analysis and exploration , 2012, ASIST.

[133]  Nicola Lettieri,et al.  Visualization of Music Plagiarism: Analysis and Evaluation , 2016, 2016 20th International Conference Information Visualisation (IV).

[134]  Luis Gustavo Nonato,et al.  MoshViz: A Detail+Overview Approach to Visualize Music Elements , 2016, IEEE Transactions on Multimedia.

[135]  Margaret Lam Beyond the score: Music visualization and digital humanities , 2012 .

[136]  Issei Fujishiro,et al.  Music learning through visualization , 2002, Second International Conference on Web Delivering of Music, 2002. WEDELMUSIC 2002. Proceedings..

[137]  Kwan-Liu Ma,et al.  Content Based Graph Visualization of Audio Data for Music Library Navigation , 2010, 2010 IEEE International Symposium on Multimedia.

[138]  Gerik Scheuermann,et al.  Interactive Visual Profiling of Musicians , 2016, IEEE Transactions on Visualization and Computer Graphics.

[139]  Marco Büchler,et al.  Visualizations for Text Re-use , 2015, 2014 International Conference on Information Visualization Theory and Applications (IVAPP).

[140]  David H. Laidlaw,et al.  Representing Uncertainty in Graph Edges: An Evaluation of Paired Visual Variables , 2015, IEEE Transactions on Visualization and Computer Graphics.

[141]  Tassie Gniady,et al.  Using Archival Texts to Create Network Graphs of Musicians in Early Modern Venice , 2017, DH.

[142]  Christopher Marks Every Noise at Once , 2020 .

[143]  Vlado Keselj,et al.  MIDIVIS: VISUALIZING MUSIC PIECES STRUCTURE VIA SIMILARITY MATRICES , 2009 .

[144]  Masataka Goto,et al.  Songrium: a music browsing assistance service based on visualization of massive open collaboration within music content creation community , 2013, OpenSym.

[145]  Bongshin Lee,et al.  Timelines Revisited: A Design Space and Considerations for Expressive Storytelling , 2017, IEEE Transactions on Visualization and Computer Graphics.

[146]  Franc Solina,et al.  Visualization of concurrent tones in music with colours , 2010, ACM Multimedia.

[147]  Lie Lu,et al.  Automatic music video generation based on temporal pattern analysis , 2004, MULTIMEDIA '04.

[148]  J. McDermid,et al.  Non-destructive investigation of aluminum alloy hemmed joints using neutron radiography and X-ray computed tomography , 2017 .

[149]  Meinard Mller,et al.  Fundamentals of Music Processing: Audio, Analysis, Algorithms, Applications , 2015 .

[150]  Elias Pampalk,et al.  Please Scroll down for Article Journal of New Music Research the Som-enhanced Jukebox: Organization and Visualization of Music Collections Based on Perceptual Models , 2022 .

[151]  Yee-Hong Yang,et al.  Music-driven character animation , 2009, TOMCCAP.

[152]  Sean M. Smith,et al.  A visualization of music , 1997, Proceedings. Visualization '97 (Cat. No. 97CB36155).

[153]  Tapio Takala,et al.  Experiments with Virtual Reality Instruments , 2005, NIME.

[154]  George Tzanetakis,et al.  Automatic Musical Genre Classification of Audio Signals , 2001, ISMIR.

[155]  Gerhard Widmer,et al.  Exploring Music Collections by Browsing Different Views , 2004, Computer Music Journal.

[156]  Christopher Raphael,et al.  Music score alignment and computer accompaniment , 2006, CACM.

[157]  Lyn Bartram,et al.  Musician Map: visualizing music collaborations over time , 2009, Electronic Imaging.

[158]  Claudio Violato,et al.  A meta‐analysis of the educational effectiveness of three‐dimensional visualization technologies in teaching anatomy , 2015, Anatomical sciences education.

[159]  Michael Weber,et al.  P.I.A.N.O.: enhancing instrument learning via interactive projected augmentation , 2013, UbiComp.

[160]  Hiroshi Ishii,et al.  MirrorFugue iii: conjuring the recorded pianist , 2013, CHI Extended Abstracts.

[161]  Elaine Chew,et al.  Visualizing Music: Tonal Progressions and Distributions , 2007, ISMIR.