Offline Handwritten Mathematical Expression Evaluator Using Convolutional Neural Network

Recognition of Offline Handwritten Mathematical Expression (HME) is a complicated task in the field of computer vision. The proposed method in this paper follows three steps: segmentation, recognition and evaluation of the HME image (which may include multiple mathematical expressions and linear equations). The segmentation of symbols from image incorporates a novel pre-contour filtration technique to remove distortions from segmented symbols. Then, recognition of segmented symbols is done using Convolutional Neural Network which is trained on an augmented dataset prepared from EMNIST and custom-built dataset giving an accuracy of 97% in recognizing the symbols correctly. Finally, the expressions/equations are evaluated by tokenizing, converting into postfix expressions and then solving using a custom-built parser.

[1]  Giang Son Tran,et al.  Handwritten Mathematical Expression Recognition Using Convolutional Neural Network , 2018, 2018 3rd International Conference on Control, Robotics and Cybernetics (CRC).

[2]  Belaid Bouikhalene,et al.  A Comparative Study of Handwritten Mathematical Symbols Recognition , 2016, 2016 13th International Conference on Computer Graphics, Imaging and Visualization (CGiV).

[3]  Gregory Cohen,et al.  EMNIST: Extending MNIST to handwritten letters , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[4]  Sagar Shinde,et al.  A new neural network based algorithm for identifying handwritten mathematical equations , 2017, 2017 International Conference on Trends in Electronics and Informatics (ICEI).

[5]  Maruska Mascarenhas,et al.  Offline Handwritten Mathematical Expression Recognition using Convolutional Neural Network , 2018, 2018 International Conference on Information , Communication, Engineering and Technology (ICICET).

[6]  Liangrui Peng,et al.  On-line Handwritten Mathematical Expression Recognition Method Based on Statistical and Semantic Analysis , 2014, 2014 11th IAPR International Workshop on Document Analysis Systems.

[7]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[8]  Bedy Purnama,et al.  Convolutional neural networks applied to handwritten mathematical symbols classification , 2016, 2016 4th International Conference on Information and Communication Technology (ICoICT).

[9]  Sagar Shinde,et al.  An improved algorithm for recognizing mathematical equations by using machine learning approach and hybrid feature extraction technique , 2017, 2017 IEEE International Conference on Electrical, Instrumentation and Communication Engineering (ICEICE).

[10]  Y. A. Joarder,et al.  Recognition and Solution for Handwritten Equation Using Convolutional Neural Network , 2018, 2018 Joint 7th International Conference on Informatics, Electronics & Vision (ICIEV) and 2018 2nd International Conference on Imaging, Vision & Pattern Recognition (icIVPR).

[11]  Masaki Nakagawa,et al.  Training an End-to-End System for Handwritten Mathematical Expression Recognition by Generated Patterns , 2017, 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR).

[12]  Harold Mouchère,et al.  Towards Handwritten Mathematical Expression Recognition , 2009, 2009 10th International Conference on Document Analysis and Recognition.