Recognizing Handwritten Mathematical Expressions as LaTex Sequences Using a Multiscale Robust Neural Network

In this paper, a robust multiscale neural network is proposed to recognize handwritten mathematical expressions and output LaTeX sequences, which can effectively and correctly focus on where each step of output should be concerned and has a positive effect on analyzing the two-dimensional structure of handwritten mathematical expressions and identifying different mathematical symbols in a long expression. With the addition of visualization, the model's recognition process is shown in detail. In addition, our model achieved 49.459% and 46.062% ExpRate on the public CROHME 2014 and CROHME 2016 datasets. The present model results suggest that the state-of-the-art model has better robustness, fewer errors, and higher accuracy.

[1]  Xiangnan He,et al.  Attentive Collaborative Filtering: Multimedia Recommendation with Item- and Component-Level Attention , 2017, SIGIR.

[2]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[3]  Yang Liu,et al.  Modeling Coverage for Neural Machine Translation , 2016, ACL.

[4]  Arun Agarwal,et al.  A Rule-Based Approach to Form Mathematical Symbols in Printed Mathematical Expressions , 2011, MIWAI.

[5]  Amit Pillay.  Intelligent Combination of Structural Analysis Algorithms: Application to Mathematical Expression Recognition , 2014 .

[6]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Jong-Hwan Kim,et al.  Mathematical Formula Recognition Based on Modified Recursive Projection Profile Cutting and Labeling with Double Linked List , 2012, RiTA.

[8]  R. Yamamoto,et al.  On-Line Recognition of Handwritten Mathematical Expression Based on Stroke-Based Stochastic Context-Free Grammar , 2006 .

[9]  Jun Du,et al.  Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition , 2018, 2018 24th International Conference on Pattern Recognition (ICPR).

[10]  George Labahn,et al.  A new approach for recognizing handwritten mathematics using relational grammars and fuzzy sets , 2013, International Journal on Document Analysis and Recognition (IJDAR).

[11]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[12]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[13]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[14]  Stéphane Lavirotte,et al.  Mathematical formula recognition using graph grammar , 1998, Electronic Imaging.

[15]  Harold Mouchère,et al.  ICFHR2016 CROHME: Competition on Recognition of Online Handwritten Mathematical Expressions , 2016, 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR).

[16]  Robert H. Anderson.  Syntax-directed recognition of hand-printed two-dimensional mathematics , 1967, Symposium on Interactive Systems for Experimental Applied Mathematics.

[17]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[18]  Yoshua Bengio,et al.  Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , 2015, ICML.

[19]  Shiliang Zhang,et al.  Watch, attend and parse: An end-to-end neural network based approach to handwritten mathematical expression recognition , 2017, Pattern Recognit..

[20]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Geoffrey E. Hinton,et al.  On the importance of initialization and momentum in deep learning , 2013, ICML.

[23]  Dit-Yan Yeung,et al.  Error detection, error correction and performance evaluation in on-line mathematical expression recognition , 2001, Pattern Recognit..

[24]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.