A system for recognizing online handwritten mathematical expressions by using improved structural analysis

A system for recognizing online handwritten mathematical expressions (MEs), by applying improved structural analysis, is proposed and experimentally evaluated on two databases. With this system, MEs are represented in the form of stochastic context-free grammar (SCFG), and the Cocke–Younger–Kasami (CYK) algorithm is used to parse two-dimensional (2D) structures of online handwritten MEs and select the best interpretation in terms of the results of symbol segmentation and recognition as well as structural analysis. A concept of “body box” is proposed, and two SVM models are applied for learning and analyzing structural relations from training patterns without the need for any heuristic decisions. Stroke order is used to reduce the complexity of the parsing algorithm. Even though SCFG does not resolve ambiguities in some cases, the proposed system still gives users a list of candidates that contains the expected result. The results of experimental evaluations of the proposed system on the CROHME 2013 and CROHME 2014 databases and on an in-house (“Hand-Math”) database show that the recognition rate of the proposed system is improved, while the processing time on a common CPU is kept to a practical level.

[1]  Masakazu Suzuki,et al.  Statistical Classification of Spatial Relationships among Mathematical Symbols , 2009, 2009 10th International Conference on Document Analysis and Recognition.

[2]  Joan-Andreu Sánchez,et al.  Classification of On-Line Mathematical Symbols with Hybrid Features and Recurrent Neural Networks , 2013, 2013 12th International Conference on Document Analysis and Recognition.

[3]  Frank K. Soong,et al.  A Unified Framework for Symbol Segmentation and Recognition of Handwritten Mathematical Expressions , 2007, Ninth International Conference on Document Analysis and Recognition (ICDAR 2007).

[4]  Harold Mouchère,et al.  A global learning approach for an online handwritten mathematical expression recognition system , 2014, Pattern Recognit. Lett..

[5]  Fotini Simistira,et al.  Recognition of online handwritten mathematical formulas using probabilistic SVMs and stochastic context free grammars , 2015, Pattern Recognit. Lett..

[6]  Robert H. Anderson.  Syntax-directed recognition of hand-printed two-dimensional mathematics , 1967, Symposium on Interactive Systems for Experimental Applied Mathematics.

[7]  Masaki Nakagawa,et al.  Objective Function Design for MCE-Based Combination of On-line and Off-line Character Recognizers for On-line Handwritten Japanese Text Recognition , 2011, 2011 International Conference on Document Analysis and Recognition.

[8]  Steve Young,et al.  Applications of stochastic context-free grammars using the Inside-Outside algorithm , 1990 .

[9]  Shi-Kuo Chang,et al.  A method for the structural analysis of two-dimensional mathematical expressions , 1970, Inf. Sci..

[10]  Catherine Lu.  Recognition of Online Handwritten Mathematical Expressions , 2013 .

[11]  Frank K. Soong,et al.  A Unified Framework for Symbol Segmentation and Recognition of Handwritten Mathematical Expressions , 2007 .

[12]  Masakazu Suzuki,et al.  Mathematical formula recognition using virtual link network , 2001, Proceedings of Sixth International Conference on Document Analysis and Recognition.

[13]  Richard Zanibbi,et al.  A shape-based layout descriptor for classifying spatial relationships in handwritten math , 2013, ACM Symposium on Document Engineering.

[14]  Stephanie Ludi,et al.  Using Off-Line Features and Synthetic Data for On-Line Handwritten Math Symbol Recognition , 2014, 2014 14th International Conference on Frontiers in Handwriting Recognition.

[15]  Joan-Andreu Sánchez,et al.  Offline Features for Classifying Handwritten Math Symbols with Recurrent Neural Networks , 2014, 2014 22nd International Conference on Pattern Recognition.

[16]  Kenneth R. Koedinger,et al.  Evaluation of multimodal input for entering mathematical equations on the computer , 2005, CHI Extended Abstracts.

[17]  Harold Mouchère,et al.  ICFHR 2014 Competition on Recognition of On-Line Handwritten Mathematical Expressions (CROHME 2014) , 2014, 2014 14th International Conference on Frontiers in Handwriting Recognition.

[18]  Jin Hyung Kim,et al.  Efficient search strategy in structural analysis for handwritten mathematical expression recognition , 2009, Pattern Recognit..

[19]  Masaki Nakagawa,et al.  A System for Recognizing Online Handwritten Mathematical Expressions and Improvement of Structure Analysis , 2014, 2014 11th IAPR International Workshop on Document Analysis Systems.

[20]  Vladimir Solmon,et al.  The estimation of stochastic context-free grammars using the Inside-Outside algorithm , 2003 .

[21]  Dit-Yan Yeung,et al.  Mathematical expression recognition: a survey , 2000, International Journal on Document Analysis and Recognition.

[22]  George Labahn,et al.  Elastic matching in linear time and constant space , 2009 .

[23]  Manfred K. Lang,et al.  A soft-decision approach for symbol segmentation within handwritten mathematical expressions , 1996, 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings.

[24]  Harold Mouchère,et al.  ICDAR 2013 CROHME: Third International Competition on Recognition of Online Handwritten Mathematical Expressions , 2013, 2013 12th International Conference on Document Analysis and Recognition.

[25]  Lei Hu,et al.  Segmenting Handwritten Math Symbols Using AdaBoost and Multi-scale Shape Context Features , 2013, 2013 12th International Conference on Document Analysis and Recognition.

[26]  Richard Zanibbi,et al.  Recognizing Mathematical Expressions Using Tree Transformation , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[27]  Robert H. Anderson.  Syntax-directed recognition of hand-printed two-dimensional mathematics , 1967, Symposium on Interactive Systems for Experimental Applied Mathematics.

[28]  Joan-Andreu Sánchez,et al.  Recognition of on-line handwritten mathematical expressions using 2D stochastic context-free grammars and hidden Markov models , 2014, Pattern Recognit. Lett..

[29]  Lei Hu,et al.  HMM-Based Recognition of Online Handwritten Mathematical Symbols Using Segmental K-Means Initialization and a Modified Pen-Up/Down Feature , 2011, 2011 International Conference on Document Analysis and Recognition.

[30]  George Labahn,et al.  A Bayesian model for recognizing handwritten mathematical expressions , 2015, Pattern Recognit..

[31]  Walter L. Ruzzo,et al.  On the Complexity of General Context-Free Language Parsing and Recognition (Extended Abstract) , 1979, ICALP.

[32]  Richard Zanibbi,et al.  Recognition and retrieval of mathematical expressions , 2011, International Journal on Document Analysis and Recognition (IJDAR).

[33]  Kenneth R. Koedinger,et al.  A paradigm for handwriting-based intelligent tutors , 2012, Int. J. Hum. Comput. Stud..

[34]  Tomoichi Takahashi,et al.  A study of symbol segmentation method for handwritten mathematical formula recognition using mathematical structure information , 2004, ICPR 2004.

[35]  George Labahn,et al.  A new approach for recognizing handwritten mathematics using relational grammars and fuzzy sets , 2013, International Journal on Document Analysis and Recognition (IJDAR).