Visualizing the Semantic Structure in Classical Music Works

A major obstacle in the appreciation of classical music is that extensive training is required to understand musical structure and compositional techniques toward comprehending the thoughts behind the musical work. In this paper, we propose an innovative visualization solution to reveal the semantic structure in classical orchestral works such that users can gain insights into musical structure and appreciate the beauty of music. We formulate the semantic structure into macrolevel layer interactions, microlevel theme variations, and macro-micro relationships between themes and layers to abstract the complicated construction of a musical composition. The visualization has been applied with success in understanding some classical music works as supported by highly promising user study results with the general audience and very positive feedback from music students and experts, demonstrating its effectiveness in conveying the sophistication and beauty of classical music to novice users with informative and intuitive displays.

[1]  Helwig Hauser,et al.  Outlier-Preserving Focus+Context Visualization in Parallel Coordinates , 2006, IEEE Transactions on Visualization and Computer Graphics.

[2]  Jörn Loviscach,et al.  Music Icons: Procedural Glyphs for Audio Files , 2006, 2006 19th Brazilian Symposium on Computer Graphics and Image Processing.

[3]  Jonathan Foote,et al.  Visualizing Musical Structure and Rhythm via Self-Similarity , 2001, ICMC.

[4]  Rumi Hiraga.  Case study: A look of performance expression , 2002, IEEE Visualization, 2002. VIS 2002..

[5]  John P. Lewis,et al.  VisualIDs: automatic distinctive icons for desktop interfaces , 2004, ACM Trans. Graph..

[6]  Jock D. Mackinlay,et al.  Cone Trees: animated 3D visualizations of hierarchical information , 1991, CHI.

[7]  Petri Toiviainen.  Visualization of tonal content with self-organizing maps and self-similarity matrices , 2005, CIE.

[8]  Colin Ware,et al.  Information Visualization: Perception for Design , 2000 .

[9]  Herman Chernoff,et al.  The Use of Faces to Represent Points in k- Dimensional Space Graphically , 1973 .

[10]  M. Wattenberg.  The Shape of Song , 2001 .

[11]  Brian Evans,et al.  Time slices to sonic maps , 2005, SIGGRAPH '05.

[12]  Karrie Karahalios,et al.  Isochords: visualizing structure in music , 2007, GI '07.

[13]  G. Kindlmann,et al.  Superquadric Glyphs for Symmetric Second-Order Tensors , 2010, IEEE Transactions on Visualization and Computer Graphics.

[14]  J. Ox.  2 performances in the 21 st. Century Virtual Color Organ: GridJam and Im Januar am Nil , 2001, Proceedings Seventh International Conference on Virtual Systems and Multimedia.

[15]  Ben Shneiderman,et al.  Readings in information visualization - using vision to think , 1999 .

[16]  M. Cooper,et al.  Revealing structure within clustered parallel coordinates displays , 2005, IEEE Symposium on Information Visualization, 2005. INFOVIS 2005..

[17]  Hans G. Kaper,et al.  Manifold Compositions, Music Visualization, and Scientific Sonification In an Immersive Virtual-reality Environment , 1998, ICMC.

[18]  Craig Stuart Sapp.  Visual hierarchical key analysis , 2005, CIE.

[19]  Adérito Marcos,et al.  An Immersive Musical Instrument Prototype , 2007, IEEE Computer Graphics and Applications.

[20]  Issei Fujishiro,et al.  Music learning through visualization , 2002, Second International Conference on Web Delivering of Music, 2002. WEDELMUSIC 2002. Proceedings..

[21]  Satoshi Tojo,et al.  ATTA: Automatic Time-Span Tree Analyzer Based on Extended GTTM , 2005, ISMIR.

[22]  Sean M. Smith,et al.  A visualization of music , 1997, Proceedings. Visualization '97 (Cat. No. 97CB36155).

[23]  Lucy T. Nowell,et al.  ThemeRiver: Visualizing Thematic Changes in Large Document Collections , 2002, IEEE Trans. Vis. Comput. Graph..

[24]  Jonathan Foote,et al.  Visualizing music and audio using self-similarity , 1999, MULTIMEDIA '99.

[25]  David Borland,et al.  Rainbow Color Map (Still) Considered Harmful , 2007, IEEE Computer Graphics and Applications.

[26]  Kevin Jennings,et al.  Hyperscore: a graphical sketchpad for novice composers , 2004, IEEE Computer Graphics and Applications.

[27]  Matthew O. Ward,et al.  Hierarchical parallel coordinates for exploration of large datasets , 1999, Proceedings Visualization '99 (Cat. No.99CB37067).

[28]  Heinrich Taube,et al.  SEE - A Structured Event Editor: Visualizing Compositional Data in Common Music , 1996, ICMC.

[29]  Carol Krumhansl,et al.  The geometry of musical structure: a brief introduction and history , 2005, CIE.

[30]  Issei Fujishiro,et al.  comp - I : 3D Visualization of MIDI Datasets , 2002 .

[31]  Rumi Hiraga,et al.  Graphical expression of the mood of music , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[32]  Daniel V. Oppenheim.  Compositional Tools for Adding Expression to Music in DMIX , 1992, ICMC.

[33]  Issei Fujishiro,et al.  Exploring MIDI datasets , 2003, SIGGRAPH '03.

[34]  Martin Wattenberg,et al.  Arc diagrams: visualizing structure in strings , 2002, IEEE Symposium on Information Visualization, 2002. INFOVIS 2002..

[35]  Craig Stuart Sapp.  Harmonic Visualizations of Tonal Music , 2001, ICMC.

[36]  ChewElaine,et al.  Interactive multi-scale visualizations of tonal evolution in MuSA.RT Opus 2 , 2005 .

[37]  Marti A. Hearst,et al.  ImproViz: visual explorations of jazz improvisations , 2005, CHI EA '05.

[38]  Steve DiPaola,et al.  Emotional remapping of music to facial animation , 2006, Sandbox '06.

[39]  Jock D. Mackinlay,et al.  The perspective wall: detail and context smoothly integrated , 1991, CHI.

[40]  Kenneth Ward Church,et al.  Dotplot : a program for exploring self-similarity in millions of lines of text and code , 1993 .

[41]  Gordon Kindlmann,et al.  Superquadric tensor glyphs , 2004, VISSYM'04.

[42]  Issei Fujishiro,et al.  The elements of graphing data , 2005, The Visual Computer.

[43]  Chaomei Chen,et al.  Information Visualization: Beyond the Horizon , 2006 .

[44]  Mark Apperley,et al.  A Bifocal Display Technique for Data Presentation , 1982, Eurographics.

[45]  Andrew R. Brown,et al.  Dynamic response: real-time adaptation for music emotion , 2005 .

[46]  Rumi Hiraga,et al.  Visualization of music performance as an aid to listener's comprehension , 2004, AVI.

[47]  Issei Fujishiro,et al.  Performance visualization: a new challenge to music through visualization , 2002, MULTIMEDIA '02.

[48]  Brian Evans.  Musical connections and heterophonic maps , 2006, SIGGRAPH '06.