Augmenting Sheet Music with Rhythmic Fingerprints

In this paper, we bridge the gap between visualization and musicology by focusing on rhythm analysis tasks, which are tedious due to the complex visual encoding of the well-established Common Music Notation (CMN). Instead of replacing the CMN, we augment sheet music with rhythmic fingerprints to mitigate the complexity originating from the simultaneous encoding of musical features. The proposed visual design exploits music theory concepts such as the rhythm tree to facilitate the understanding of rhythmic information. Juxtaposing sheet music and the rhythmic fingerprints maintains the connection to the familiar representation. To investigate the usefulness of the rhythmic fingerprint design for identifying and comparing rhythmic patterns, we conducted a controlled user study with four experts and four novices. The results show that the rhythmic fingerprints enable novice users to recognize rhythmic patterns that only experts can identify using non-augmented sheet music.

[1]  H. C. Longuet-Higgins The perception of music. , 1979, Proceedings of the Royal Society of London. Series B, Biological sciences.

[2]  Michael Good MusicXML: An internet-friendly format for sheet music , 2001 .

[3]  Petra Isenberg,et al.  Evaluation of alternative glyph designs for time series data in a small multiple setting , 2013, CHI.

[4]  Frank Maurer,et al.  A generative layout approach for rooted tree drawings , 2013, 2013 IEEE Pacific Visualization Symposium (PacificVis).

[5]  Godfried T. Toussaint,et al.  Mathematical Notation, Representation, and Visualization of Musical Rhythm: A Comparative Perspective , 2012 .

[6]  Godfried T. Toussaint,et al.  Rhythm Complexity Measures: A Comparison of Mathematical Models of Human Perception and Performance , 2008, ISMIR.

[7]  Hope R Strayer,et al.  From Neumes to Notes: The Evolution of Music Notation , 2013 .

[8]  Michael Burch,et al.  Space-Reclaiming Icicle Plots , 2020, 2020 IEEE Pacific Visualization Symposium (PacificVis).

[9]  Greta Franzini,et al.  On Close and Distant Reading in Digital Humanities: A Survey and Future Challenges , 2015, EuroVis.

[10]  Werner Breig Bachs Goldberg-Variationen als zyklisches Werk , 1975 .

[11]  Cecilia K. Hultberg Approaches to Music Notation: The printed score as a mediator of meaning in Western tonal tradition , 2002 .

[12]  Daniel A. Keim,et al.  Analyzing Visual Mappings of Traditional and Alternative Music Notation , 2018, ArXiv.

[13]  Cheryl Z. Qian,et al.  Comparison of Circle and Dodecagon Clock Designs for Visualizing 24-Hour Cyclical Data , 2017, HCI.

[14]  John W. Scheib,et al.  Takadimi: A Rhythm System for All Ages , 2006 .

[15]  Winnie Wing-Yi Chan A Report on Musical Structure Visualization , 2007 .

[16]  Tatiana Bershadskaya Analogies and Parallels in the Structure of Music and Verbal Languages , 2015, LMAC.

[17]  Delfina Malandrino,et al.  Visualization and Music Harmony: Design, Implementation, and Evaluation , 2018, 2018 22nd International Conference Information Visualisation (IV).

[18]  Dharmesh Patel Rhythm , 1919, The Craft of Poetry.

[19]  Jeffrey Heer,et al.  SpanningAspectRatioBank Easing FunctionS ArrayIn ColorIn Date Interpolator MatrixInterpola NumObjecPointI Rectang ISchedu Parallel Pause Scheduler Sequen Transition Transitioner Transiti Tween Co DelimGraphMLCon IData JSONCon DataField DataSc Dat DataSource Data DataUtil DirtySprite LineS RectSprite , 2011 .

[20]  Mennatallah El-Assady,et al.  Augmenting Music Sheets with Harmonic Fingerprints , 2019, DocEng.

[21]  Iris Vessey,et al.  Cognitive Fit: A Theory‐Based Analysis of the Graphs Versus Tables Literature* , 1991 .

[22]  Godfried T. Toussaint,et al.  Measuring musical rhythm similarity : Transformation versus feature-based methods , 2013 .

[23]  Ben Shneiderman,et al.  Tree-maps: a space-filling approach to the visualization of hierarchical information structures , 1991, Proceeding Visualization '91.

[24]  Simon Dixon,et al.  A Review of Automatic Rhythm Description Systems , 2005, Computer Music Journal.

[25]  Stefan Jänicke,et al.  A Survey on Visualizations for Musical Data , 2020, Comput. Graph. Forum.

[26]  Meinard Müller,et al.  Music Structure Analysis , 2021, Fundamentals of Music Processing.

[27]  C. Krumhansl Rhythm and pitch in music cognition. , 2000, Psychological bulletin.

[28]  Godfried T. Toussaint,et al.  Computational geometric aspects of rhythm, melody, and voice-leading , 2010, Comput. Geom..

[29]  Bruce Dalby Toward an Effective Pedagogy for Teaching Rhythm: Gordon and Beyond , 2005 .

[30]  H. C. Losguet-Higgins The Perception of Music , 1978 .

[31]  Jieping Xu,et al.  Rhythm-Based Segmentation of Popular Chinese Music , 2005, ISMIR.

[32]  Meinard Müller Tempo and Beat Tracking , 2015 .

[33]  Mathieu Giraud,et al.  RHYTHM EXTRACTION FROM POLYPHONIC SYMBOLIC MUSIC , 2011, ISMIR 2011.

[34]  K. Swanwick,et al.  Musical Knowledge: Intuition, analysis and music education , 1994 .

[35]  Richard F. Riesenfeld,et al.  A Survey of Radial Methods for Information Visualization , 2009, IEEE Transactions on Visualization and Computer Graphics.

[36]  William A. Sethares,et al.  Rhythm and Transforms , 2007 .

[37]  Gerik Scheuermann,et al.  Enhancing Close Reading , 2016, DH.

[38]  V. Hashemi,et al.  Inducing Meter for a Rhythm and Measures for Rhythm Complexity , 2008 .

[39]  Christopher Ariza,et al.  Music21: A Toolkit for Computer-Aided Musicology and Symbolic Music Data , 2010, ISMIR.

[40]  J. B. Kruskal,et al.  Icicle Plots: Better Displays for Hierarchical Clustering , 1983 .

[41]  Masataka Goto,et al.  Music Structure Analysis from Acoustic Signals , 2008 .

[42]  Deborah L. McGuinness,et al.  A Music Theory Ontology , 2018, SAAM@ISWC.

[43]  O F Herrero Symmetry in music , 2010 .

[44]  Raúl Coré,et al.  Grove Music Online , 2006 .

[45]  Godfried T. Toussaint,et al.  The Geometry of Musical Rhythm , 2013, JCDCG.

[46]  Jeffrey Heer,et al.  D³ Data-Driven Documents , 2011, IEEE Transactions on Visualization and Computer Graphics.

[47]  K. Stone Problems and Methods of Notation , 1963 .

[48]  J. Stasko,et al.  Focus+context display and navigation techniques for enhancing radial, space-filling hierarchy visualizations , 2000, IEEE Symposium on Information Visualization 2000. INFOVIS 2000. Proceedings.

[49]  R. Jackendoff,et al.  A Generative Theory of Tonal Music , 1985 .