Optical music recognition: state-of-the-art and open issues

For centuries, music has been shared and remembered by two traditions: aural transmission and in the form of written documents normally called musical scores. Many of these scores exist in the form of unpublished manuscripts and hence they are in danger of being lost through the normal ravages of time. To preserve the music some form of typesetting or, ideally, a computer system that can automatically decode the symbolic images and create new scores is required. Programs analogous to optical character recognition systems called optical music recognition (OMR) systems have been under intensive development for many years. However, the results to date are far from ideal. Each of the proposed methods emphasizes different properties and therefore makes it difficult to effectively evaluate its competitive advantages. This article provides an overview of the literature concerning the automatic analysis of images of printed and handwritten musical scores. For self-containment and for the benefit of the reader, an introduction to OMR processing systems precedes the literature overview. The following study presents a reference scheme for any researcher wanting to compare new OMR algorithms against well-known ones.

[1]  Ichiro Fujinaga,et al.  A Comparative Survey of Image Binarisation Algorithms for Optical Recognition on Degraded Musical Sources , 2007, ISMIR.

[2]  Jean Camillerapp,et al.  A way to separate knowledge from program in structured document analysis: application to optical music recognition , 1995, Proceedings of 3rd International Conference on Document Analysis and Recognition.

[3]  Jaime S. Cardoso,et al.  Robust Staffline Thickness and Distance Estimation in Binary and Gray-Level Music Scores , 2010, 2010 20th International Conference on Pattern Recognition.

[4]  Jim R. Parker,et al.  Automatic computer recognition of printed music , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[5]  T. W. Ridler,et al.  Picture thresholding using an iterative selection method. , 1978 .

[6]  Andrew K. C. Wong,et al.  A new method for gray-level picture thresholding using the entropy of the histogram , 1985, Comput. Vis. Graph. Image Process..

[7]  Christoph Dalitz,et al.  Reject Options and Confidence Measures for kNN Classifiers , 2009 .

[8]  Henry S. Baird,et al.  A Critical Survey of Music Image Analysis , 1992 .

[9]  Carlos Guedes,et al.  A Method for Music Symbols Extraction based on Musical Rules , 2011 .

[10]  M. Ibrahim Sezan,et al.  A Peak Detection Algorithm and its Application to Histogram-Based Image Data Reduction , 1990, Comput. Vis. Graph. Image Process..

[11]  Alberto Ciampa,et al.  On Automatic Pattern Recognition and Acquisition of Printed Music , 1982, ICMC.

[12]  Carlos Guedes,et al.  Integrated Recognition System for Music Scores , 2008, ICMC.

[13]  Du-Ming Tsai,et al.  A fast thresholding selection procedure for multimodal and unimodal histograms , 1995, Pattern Recognit. Lett..

[14]  Laurent Pugin,et al.  Optical Music Recognition of Early Typographic Prints using Hidden Markov Models , 2006 .

[15]  Carlos Guedes,et al.  A connected path approach for staff detection on a music score , 2008, 2008 15th IEEE International Conference on Image Processing.

[16]  Kia Ng Optical Music Analysis for Printed Music Score and Handwritten Music Manuscript , 2004 .

[17]  Meinard Müller,et al.  Automatic Mapping of Scanned Sheet Music to Audio Recordings , 2008, ISMIR.

[18]  Meinard Müller,et al.  Automated Synchronization of Scanned Sheet Music with Audio Recordings , 2007, ISMIR.

[19]  Donald Byrd,et al.  Prospects for Improving OMR with Multiple Recognizers , 2006, ISMIR.

[20]  Carlos Guedes,et al.  Staff Detection with Stable Paths , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Ichiro Fujinaga Exemplar-based Learning in Adaptive Optical Music Recognition System , 1996, ICMC.

[22]  Øivind Due Trier,et al.  Evaluation of Binarization Methods for Document Images , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Zhang Wenjun,et al.  Pick-up the Musical Information from Digital Musical Score Based on Mathematical Morphology and Music Notation , 2009, 2009 First International Workshop on Education Technology and Computer Science.

[24]  Nailja Luth,et al.  Automatic identification of music notations , 2002, Second International Conference on Web Delivering of Music, 2002. WEDELMUSIC 2002. Proceedings..

[25]  Pierfrancesco Bellini,et al.  Assessing Optical Music Recognition Tools , 2007, Computer Music Journal.

[26]  Prasanna K. Sahoo,et al.  Threshold selection using Renyi's entropy , 1997, Pattern Recognit..

[27]  Anil K. Jain,et al.  Goal-Directed Evaluation of Binarization Methods , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Jaime S. Cardoso,et al.  Optical recognition of music symbols , 2010, International Journal on Document Analysis and Recognition (IJDAR).

[29]  M. Szwoch Guido: A Musical Score Recognition System , 2007 .

[30]  S. D. Yanowitz,et al.  A new method for image segmentation , 1988, [1988 Proceedings] 9th International Conference on Pattern Recognition.

[31]  Timothy C. Bell,et al.  The Challenge of Optical Music Recognition , 2001, Comput. Humanit..

[32]  Isabel Barbancho,et al.  Optical Music Recognition for Scores Written in White Mensural Notation , 2009, EURASIP J. Image Video Process..

[33]  Wayne Nilback An introduction to digital image processing , 1985 .

[34]  Yves Grandvalet,et al.  Support Vector Machines with a Reject Option , 2008, NIPS.

[35]  Ichiro Fujinaga,et al.  Optical Music Interpretation , 2002, SSPR/SPR.

[36]  Michael Kassler,et al.  Optical Character-Recognition of Printed Music: A Review of Two Dissertations@@@Automatic Recognition of Sheet Music@@@Computer Pattern Recognition of Standard Engraved Music Notation , 1972 .

[37]  Pierfrancesco Bellini,et al.  Optical music sheet segmentation , 2001, Proceedings First International Conference on WEB Delivering of Music. WEDELMUSIC 2001.

[38]  A. D. Brink,et al.  Minimum cross-entropy threshold selection , 1996, Pattern Recognit..

[39]  Yasuaki Nakano,et al.  Note Symbol Extraction for Printed Piano Scores Using Neural Networks (Special Issue on Character Recognition and Document Understanding) , 1996 .

[40]  Ivan Bruno,et al.  Optical Music Imaging: Music Document Digitisation, Recognition, Evaluation, and Restoration , 2008 .

[41]  Roland Göcke Building a system for writer identification on handwritten music scores , 2003 .

[42]  Mao-Jiun J. Wang,et al.  Image thresholding by minimizing the measures of fuzzines , 1995, Pattern Recognit..

[43]  Alicia Fornés,et al.  On the Use of Textural Features for Writer Identification in Old Handwritten Music Scores , 2009, 2009 10th International Conference on Document Analysis and Recognition.

[44]  Nial Friel,et al.  A new thresholding technique based on random sets , 1999, Pattern Recognit..

[45]  Bülent Sankur,et al.  Survey over image thresholding techniques and quantitative performance evaluation , 2004, J. Electronic Imaging.

[46]  David Bainbridge,et al.  Extensible optical music recognition , 1997 .

[47]  E. Carrapatoso,et al.  A Shortest Path Approach for Staff Line Detection , 2007, Third International Conference on Automated Production of Cross Media Content for Multi-Channel Distribution (AXMEDIS'07).

[48]  Marcin Luckner,et al.  Automatic Knowledge Acquisition: Recognizing Music Notation with Methods of Centroids and Classifications Trees , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[49]  Ichiro Fujinaga,et al.  A Comparative Study of Staff Removal Algorithms , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[50]  N. Otsu A threshold selection method from gray level histograms , 1979 .

[51]  O. Jenkins,et al.  MusicHand : A Handwritten Music Recognition System , 2005 .

[52]  Pheng-Ann Heng,et al.  A double-threshold image binarization method based on edge detector , 2008, Pattern Recognit..

[53]  Ichiro Fujinaga,et al.  Staff Detection and Removal , 2004 .

[54]  Jacques D. Fleuriot,et al.  Diagrammatically-Driven Formal Verification of Web-Services Composition , 2012, Diagrams.

[55]  Adnan Khashman,et al.  A Novel Thresholding Method for Text Separation and Document Enhancement , 2007 .

[56]  Mariusz Szwoch,et al.  Using MusicXML to Evaluate Accuracy of OMR Systems , 2008, Diagrams.

[57]  Márcio Portes de Albuquerque,et al.  Image thresholding using Tsallis entropy , 2004, Pattern Recognit. Lett..

[58]  Ichiro Fujinaga,et al.  Gamera: Optical music recognition in a new shell , 2002, ICMC.

[59]  Wayne Niblack,et al.  An introduction to digital image processing , 1986 .

[60]  Ioannis Pratikakis,et al.  An Adaptive Binarization Technique for Low Quality Historical Documents , 2004, Document Analysis Systems.

[61]  J. W. Roach,et al.  Using domain knowledge in low-level visual processing to interpret handwritten music: An experiment , 1988, Pattern Recognit..

[62]  Mariusz Szwoch,et al.  A Robust Detector for Distorted Music Staves , 2005, CAIP.

[63]  Anil K. Jain,et al.  Representation and Recognition of Handwritten Digits Using Deformable Templates , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[64]  Sylvie Philipp-Foliguet,et al.  Printed music recognition , 1993, Proceedings of 2nd International Conference on Document Analysis and Recognition (ICDAR '93).

[65]  Gilson A. Giraldi,et al.  Music Score Binarization Based on Domain Knowledge , 2011, IbPRIA.

[66]  Bertrand Coüasnon,et al.  Segmentation et reconnaissance de documents guidees par la connaissance a priori : application aux partitions musicales , 1996 .

[67]  N. B. Venkateswarlu Implementation of some image thresholding algorithms on a connection machine-200 , 1995, Pattern Recognit. Lett..

[68]  Yan Chen,et al.  Comparison of some thresholding algorithms for text/background segmentation in difficult document images , 2003, Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings..

[69]  David Cooper,et al.  Embracing the Composer: Optical Recognition of Handwritten Manuscripts , 1999, International Conference on Mathematics and Computing.

[70]  Nicholas Paul Carter Automatic recognition of printed music in the context of electronic publishing , 1989 .

[71]  Ivan Bruno,et al.  Optical Music Recognition: Architecture and Algorithms , 2007 .

[72]  Laurent Pugin,et al.  Optical Music Recognitoin of Early Typographic Prints using Hidden Markov Models , 2006, ISMIR.

[73]  Susan Ella George,et al.  Visual Perception of Music Notation: On-Line and Off-Line Recognition , 2004 .

[74]  Kenji Shoji,et al.  Symbol Recognition of Printed Piano Scores with Touching Symbols , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[75]  Eric Nichols,et al.  Lyric Extraction and Recognition on Digital Images of Early Music Sources , 2009, ISMIR.

[76]  Masayuki Okamoto,et al.  Stave Extraction for Printed Music Scores Using DP Matching , 2004, J. Adv. Comput. Intell. Intell. Informatics.

[77]  Meinard Müller,et al.  Multimodal presentation and browsing of music , 2008, ICMI '08.

[78]  Kim C. Ng,et al.  Recognition and reconstruction of primitives in music scores , 1996, Image Vis. Comput..

[79]  C. Brisset Using Logic Programming Languages For Optical Music Recognition , 1995 .

[80]  Ichlro FuJinaga,et al.  Optical Music Recognition Using Projections , 1988 .

[81]  Jean Camillerapp,et al.  A robust detector for music staves , 1993, Proceedings of 2nd International Conference on Document Analysis and Recognition (ICDAR '93).

[82]  Alicia Fornés,et al.  Primitive Segmentation in Old Handwritten Music Scores , 2005, GREC.

[83]  Donald Byrd,et al.  Towards Musicdiff: A Foundation for Improved Optical Music Recognition Using Multiple Recognizers , 2007, ISMIR.

[84]  W. Homenda Optical Music Recognition: the Case Study of Pattern Recognition , 2005, CORES.

[85]  Umapada Pal,et al.  An Efficient Staff Removal Approach from Printed Musical Documents , 2010, 2010 20th International Conference on Pattern Recognition.

[86]  Ana Rebelo New methodologies towards an automatic optical recognition of handwritten musical scores , 2009 .

[87]  Timothy C. Bell,et al.  A music notation construction engine for optical music recognition , 2003, Softw. Pract. Exp..

[88]  Sergio Escalera,et al.  Handwritten Symbol Recognition by a Boosted Blurred Shape Model with Error Correction , 2007, IbPRIA.

[89]  Ichiro Fujinaga,et al.  Goal-directed evaluation for the improvement of optical music recognition on early music prints , 2007, JCDL '07.

[90]  Kazuhiko Yamamoto,et al.  Structured Document Image Analysis , 1992, Springer Berlin Heidelberg.

[91]  A. Khashman,et al.  Novel Thresholding Method for Document Analysis , 2006, 2006 IEEE International Conference on Industrial Technology.

[92]  Isabelle Bloch,et al.  Robust and Adaptive OMR System Including Fuzzy Modeling, Fusion of Musical Rules, and Possible Error Detection , 2007, EURASIP J. Adv. Signal Process..

[93]  Francisco Escolano,et al.  Proceedings of the Joint IAPR International Workshop on Structural, Syntactic, and Statistical Pattern Recognition - Volume 8621 , 2014 .

[94]  Susan E. George Lyric Recognition and Christian Music , 2004 .

[95]  Ichiro Fujinaga,et al.  USING HIDDEN MARKOV MODELS , 2007 .

[96]  Ichiro Fujinaga,et al.  Recommended best practices for digital image capture of musical scores , 2003, OCLC Syst. Serv..

[97]  Gardner Read,et al.  Music notation;: A manual of modern practice , 1972 .

[98]  Fabio Roli,et al.  Proceedings of the 2006 joint IAPR international conference on Structural, Syntactic, and Statistical Pattern Recognition , 2002 .

[99]  Alicia Fornés,et al.  Writer Identification in Old Handwritten Music Scores , 2008, 2008 The Eighth IAPR International Workshop on Document Analysis Systems.

[100]  José Manuel Rebordão,et al.  An amplitude segmentation method based on the distribution function of an image , 1984, Comput. Vis. Graph. Image Process..

[101]  Jaime S. Cardoso,et al.  An Ordinal Data Method for the Classification with Reject Option , 2009, 2009 International Conference on Machine Learning and Applications.

[102]  João Leite,et al.  Hypothetical Reasoning : an application to OpticalMusic , 1999 .

[103]  Ichiro Fujinaga,et al.  Optical Music Recognition System within a Large-Scale Digitization Project , 2000, ISMIR.