Multi-Language Online Handwriting Recognition

We describe Google's online handwriting recognition system that currently supports 22 scripts and 97 languages. The system's focus is on fast, high-accuracy text entry for mobile, touch-enabled devices. We use a combination of state-of-the-art components and combine them with novel additions in a flexible framework. This architecture allows us to easily transfer improvements between languages and scripts. This made it possible to build recognizers for languages that, to the best of our knowledge, are not handled by any other online handwriting recognition system. The approach also enabled us to use the same architecture both on very powerful machines for recognition in the cloud as well as on mobile devices with more limited computational power by changing some of the settings of the system. In this paper we give a general overview of the system architecture and the novel components, such as unified time- and position-based input interpretation, trainable segmentation, minimum-error rate training for feature combination, and a cascade of pruning strategies. We present experimental results for different setups. The system is currently publicly available in several Google products, for example in Google Translate and as an input method for Android devices.

[1]  Sargur N. Srihari,et al.  On-Line and Off-Line Handwriting Recognition: A Comprehensive Survey , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Guy Jacobson,et al.  Space-efficient static trees and graphs , 1989, 30th Annual Symposium on Foundations of Computer Science.

[3]  Sanjiv Kumar,et al.  Nearest Neighbor Search in Google Correlate , 2013 .

[4]  Umapada Pal,et al.  Touching numeral segmentation using water reservoir concept , 2003, Pattern Recognit. Lett..

[5]  George Nagy,et al.  Twenty Years of Document Image Analysis in PAMI , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  John Bennett,et al.  The effect of large training set sizes on online Japanese Kanji and English cursive recognizers , 2002, Proceedings Eighth International Workshop on Frontiers in Handwriting Recognition.

[7]  Andreas Stolcke,et al.  SRILM - an extensible language modeling toolkit , 2002, INTERSPEECH.

[8]  Isabelle Guyon,et al.  UNIPEN project of on-line data exchange and recognizer benchmarks , 1994, Proceedings of the 12th IAPR International Conference on Pattern Recognition, Vol. 3 - Conference C: Signal Processing (Cat. No.94CH3440-5).

[9]  Robert L. Mercer,et al.  The Mathematics of Statistical Machine Translation: Parameter Estimation , 1993, CL.

[10]  Thorsten Brants,et al.  Large Language Models in Machine Translation , 2007, EMNLP.

[11]  Wolfgang Macherey,et al.  Lattice-based Minimum Error Rate Training for Statistical Machine Translation , 2008, EMNLP.

[12]  Eugene H. Ratzlaff.  Methods, reports and survey for the comparison of diverse isolated character recognition results on the UNIPEN database , 2003, Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings..

[13]  David Goldberg,et al.  Touch-typing with a stylus , 1993, INTERCHI.

[14]  James A. Pittman,et al.  Handwriting Recognition: Tablet PC Text Input , 2007, Computer.

[15]  Xiang-Dong Zhou,et al.  Online Handwritten Japanese Character String Recognition Incorporating Geometric Context , 2007 .

[16]  Jin Hyung Kim,et al.  Online Handwriting Recognition , 2014, Handbook of Document Image Processing and Recognition.

[17]  Éric Anquetil,et al.  HBF49 feature set: A first unified baseline for online symbol recognition , 2013, Pattern Recognit..

[18]  Sriganesh Madhvanath,et al.  Hidden Markov Models for Online Handwritten Tamil Word Recognition , 2007 .

[19]  Hermann Ney,et al.  Deformation Models for Image Recognition , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Gerhard Rigoll,et al.  Novel Hybrid NN/HMM Modelling Techniques for On-line Handwriting Recognition , 2006 .

[21]  Andrew Zisserman,et al.  Three things everyone should know to improve object retrieval , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Masaki Nakagawa,et al.  The state of the art in Japanese online handwriting recognition compared to techniques in western handwriting recognition , 2003, Document Analysis and Recognition.

[23]  Fumitaka Kimura,et al.  Modified Quadratic Discriminant Functions and the Application to Chinese Character Recognition , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Marc'Aurelio Ranzato,et al.  Large Scale Distributed Deep Networks , 2012, NIPS.

[25]  Fadi Biadsy,et al.  Segmentation-Free Online Arabic Handwriting Recognition , 2011, Int. J. Pattern Recognit. Artif. Intell..

[26]  Ching Y. Suen,et al.  The State of the Art in Online Handwriting Recognition , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[27]  Thomas Deselaers,et al.  GyroPen: Gyroscopes for Pen-Input With Mobile Phones , 2015, IEEE Transactions on Human-Machine Systems.

[28]  Richard F. Lyon,et al.  Combining Neural Networks and Context-Driven Search for On-Line, Printed Handwriting Recognition in the Newton , 1996, Neural Networks: Tricks of the Trade.

[29]  Marcus Liwicki,et al.  Combining diverse systems for handwritten text line recognition , 2009, Machine Vision and Applications.

[30]  Zhen-Long Bai,et al.  A study on the use of 8-directional features for online handwritten Chinese character recognition , 2005, Eighth International Conference on Document Analysis and Recognition (ICDAR'05).

[31]  Alexander H. Waibel,et al.  Online handwriting recognition: the NPen++ recognizer , 2001, International Journal on Document Analysis and Recognition.

[32]  Claus Bahlmann,et al.  The writer independent online handwriting recognition system frog on hand and cluster generative statistical dynamic time warping , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Patrice Y. Simard,et al.  Best practices for convolutional neural networks applied to visual document analysis , 2003, Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings..

[34]  Masaki Nakagawa,et al.  'Online recognition of Chinese characters: the state-of-the-art , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Giovanni Seni,et al.  Large Vocabulary Recognition of On-Line Handwritten Cursive Words , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[37]  Richard M. Schwartz,et al.  An Omnifont Open-Vocabulary OCR System for English and Arabic , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[38]  Marcus Liwicki,et al.  IAM-OnDB - an on-line English sentence database acquired from handwritten text on a whiteboard , 2005, Eighth International Conference on Document Analysis and Recognition (ICDAR'05).

[39]  David J. Hand,et al.  Classifier Technology and the Illusion of Progress , 2006, math/0606441.