Towards an IMU-based Pen Online Handwriting Recognizer

Most online handwriting recognition systems require the use of specific writing surfaces to extract positional data. In this paper we present a online handwriting recognition system for word recognition which is based on inertial measurement units (IMUs) for digitizing text written on paper. This is obtained by means of a sensor-equipped pen that provides acceleration, angular velocity, and magnetic forces streamed via Bluetooth. Our model combines convolutional and bidirectional LSTM networks, and is trained with the Connectionist Temporal Classification loss that allows the interpretation of raw sensor data into words without the need of sequence segmentation. We use a dataset of words collected using multiple sensor-enhanced pens and evaluate our model on distinct test sets of seen and unseen words achieving a character error rate of 17.97% and 17.08%, respectively, without the use of a dictionary or language model.

[1]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[2]  Surbhi Mishra,et al.  Online and offline character recognition: A survey , 2016, 2016 International Conference on Communication and Signal Processing (ICCSP).

[3]  A. Graves,et al.  Unconstrained Online Handwriting Recognition with Recurrent Neural Networks , 2007 .

[4]  Marcus Liwicki,et al.  A novel approach to on-line handwriting recognition based on bidirectional long short-term memory networks , 2007 .

[5]  Marcus Liwicki,et al.  Combining diverse systems for handwritten text line recognition , 2009, Machine Vision and Applications.

[7]  Yi Zhang,et al.  End-to-end speech recognition system based on improved CLDNN structure , 2019, 2019 IEEE 8th Joint International Information Technology and Artificial Intelligence Conference (ITAIC).

[8]  Tara N. Sainath,et al.  Convolutional, Long Short-Term Memory, fully connected Deep Neural Networks , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[9]  Michael Rohs,et al.  Pentelligence: Combining Pen Tip Motion and Writing Sounds for Handwritten Digit Recognition , 2018, CHI.

[10]  Masaki Nakagawa,et al.  'Online recognition of Chinese characters: the state-of-the-art , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Victor Carbune,et al.  Multi-Language Online Handwriting Recognition , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Patrick Shen-Pei Wang,et al.  Handwritten Bank Check Recognition of Courtesy Amounts , 2004, Int. J. Image Graph..

[13]  Amarjot Singh,et al.  A Survey of OCR Applications , 2012 .

[14]  Masaki Nakagawa,et al.  The state of the art in Japanese online handwriting recognition compared to techniques in western handwriting recognition , 2003, Document Analysis and Recognition.

[15]  Weiqiang Wang,et al.  In-air handwritten English word recognition using attention recurrent translator , 2017, Neural Computing and Applications.

[16]  Sargur N. Srihari,et al.  On-Line and Off-Line Handwriting Recognition: A Comprehensive Survey , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  Jürgen Schmidhuber,et al.  Framewise phoneme classification with bidirectional LSTM and other neural network architectures , 2005, Neural Networks.

[18]  Mohamed Elhafiz Mustafa,et al.  HMM based approach for Online Arabic Handwriting recognition , 2014, 2014 14th International Conference on Intelligent Systems Design and Applications.

[19]  Sargur N. Srihari,et al.  Recognition of handwritten and machine-printed text for postal address interpretation , 1993, Pattern Recognit. Lett..

[20]  Weiqiang Wang,et al.  A Unified CNN-RNN Approach for in-Air Handwritten English Word Recognition , 2018, 2018 IEEE International Conference on Multimedia and Expo (ICME).

[21]  Paavo Alku,et al.  ICASSP The IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2011 .

[22]  Jeen-Shing Wang,et al.  An Accelerometer-Based Digital Pen With a Trajectory Recognition Algorithm for Handwritten Digit and Gesture Recognition , 2012, IEEE Transactions on Industrial Electronics.

[23]  Zhe-Ting Liu,et al.  An Imu-Based Wearable Ring For On-Surface Handwriting Recognition , 2020, 2020 International Symposium on VLSI Design, Automation and Test (VLSI-DAT).

[24]  Yoshua Bengio,et al.  LeRec: A NN/HMM Hybrid for On-Line Handwriting Recognition , 1995, Neural Computation.

[25]  S. R. Mahadeva Prasanna,et al.  Exploration of CNN Features for Online Handwriting Recognition , 2019, 2019 International Conference on Document Analysis and Recognition (ICDAR).

[26]  Redouane Tlemsani,et al.  An Improved Arabic On-Line Characters Recognition System , 2018, 2018 International Arab Conference on Information Technology (ACIT).

[27]  Wang Jeen-Shing,et al.  Online Handwriting Recognition Using an Accelerometer-Based Pen Device , 2013, CSE 2013.

[28]  OttFelix,et al.  The OnHW Dataset , 2020 .

[29]  Alexander H. Waibel,et al.  Online handwriting recognition: the NPen++ recognizer , 2001, International Journal on Document Analysis and Recognition.

[30]  Isabelle Guyon,et al.  UNIPEN project of on-line data exchange and recognizer benchmarks , 1994, Proceedings of the 12th IAPR International Conference on Pattern Recognition, Vol. 3 - Conference C: Signal Processing (Cat. No.94CH3440-5).

[31]  Ifeyinwa E. Achumba,et al.  Sensor Data Acquisition and Processing Parameters for Human Activity Classification , 2014, Sensors.

[32]  Bjoern Eskofier,et al.  Digitizing Handwriting with a Sensor Pen: A Writer-Independent Recognizer , 2020, 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR).

[33]  Victor Carbune,et al.  Fast multi-language LSTM-based online handwriting recognition , 2020, International Journal on Document Analysis and Recognition (IJDAR).

[34]  Marc Kurz,et al.  What Did You Mean? An Evaluation of Online Character Recognition Approaches , 2019, 2019 International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob).

[35]  Martin H. Fischer,et al.  Is Handwriting Performance Affected by the Writing Surface? Comparing Preschoolers', Second Graders', and Adults' Writing Performance on a Tablet vs. Paper , 2016, Front. Psychol..

[36]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[37]  Marcus Liwicki,et al.  IAM-OnDB - an on-line English sentence database acquired from handwritten text on a whiteboard , 2005, Eighth International Conference on Document Analysis and Recognition (ICDAR'05).

[38]  Jürgen Schmidhuber,et al.  Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks , 2006, ICML.

[39]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[40]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[41]  Jin Hyung Kim,et al.  Online Handwriting Recognition , 2014, Handbook of Document Image Processing and Recognition.

[42]  Tanja Schultz,et al.  Airwriting: Hands-Free Mobile Text Input by Spotting and Continuous Recognition of 3d-Space Handwriting with Inertial Sensors , 2012, 2012 16th International Symposium on Wearable Computers.

[43]  Richard F. Lyon,et al.  Combining Neural Networks and Context-Driven Search for Online, Printed Handwriting Recognition in the NEWTON , 1998, AI Mag..