Effects of Language Modeling and its Personalization on Touchscreen Typing Performance

Modern smartphones correct typing errors and learn user-specific words (such as proper names). Both techniques are useful, yet little has been published about their technical specifics and concrete benefits. One reason is that typing accuracy is difficult to measure empirically on a large scale. We describe a closed-loop, smart touch keyboard (STK) evaluation system that we have implemented to solve this problem. It includes a principled typing simulator for generating human-like noisy touch input, a simple-yet-effective decoder for reconstructing typed words from such spatial data, a large web-scale background language model (LM), and a method for incorporating LM personalization. Using the Enron email corpus as a personalization test set, we show for the first time at this scale that a combined spatial-language model reduces word error rate from a pre-model baseline of 38.4% down to 5.7%, and that LM personalization can improve this further to 4.6%.

[1]  Jimmy J. Lin,et al.  Single-document and multi-document summarization techniques for email threads using sentence compression , 2008, Inf. Process. Manag..

[2]  Slava M. Katz,et al.  Estimation of probabilities from sparse data for the language model component of a speech recognizer , 1987, IEEE Trans. Acoust. Speech Signal Process..

[3]  Frederick Reiss,et al.  Domain Adaptation of Rule-Based Annotators for Named-Entity Recognition Tasks , 2010, EMNLP.

[4]  Patrick Baudisch,et al.  Understanding touch , 2011, CHI.

[5]  Jerome R. Bellegarda,et al.  Statistical language model adaptation: review and perspectives , 2004, Speech Commun..

[6]  Marcello Federico,et al.  Language Model Adaptation , 1999 .

[7]  Per Ola Kristensson,et al.  Performance comparisons of phrase sets and presentation styles for text entry evaluations , 2012, IUI '12.

[8]  Giuseppe Carenini,et al.  Domain Adaptation to Summarize Human Conversations , 2010 .

[9]  I. Scott MacKenzie,et al.  Text Entry for Mobile Computing: Models and Methods,Theory and Practice , 2002, Hum. Comput. Interact..

[10]  Sidney L. Smith,et al.  Alphabetic Data Entry via the Touch-Tone Pad: A Comment , 1971 .

[11]  Karen Spärck Jones.  Towards Better NLP System Evaluation , 1994, HLT.

[12]  Hermann Ney,et al.  Improved backing-off for M-gram language modeling , 1995, 1995 International Conference on Acoustics, Speech, and Signal Processing.

[13]  Shumin Zhai,et al.  The word-gesture keyboard: reimagining keyboard interaction , 2012, CACM.

[14]  Thorsten Brants,et al.  Study on interaction between entropy pruning and kneser-ney smoothing , 2010, INTERSPEECH.

[15]  Per Ola Kristensson,et al.  A versatile dataset for text entry evaluations based on genuine mobile emails , 2011, Mobile HCI.

[16]  Alan F. Blackwell,et al.  Dasher—a data entry interface using continuous gestures and language models , 2000, UIST '00.

[17]  Shumin Zhai,et al.  Shorthand writing on stylus keyboard , 2003, CHI '03.

[18]  Shumin Zhai,et al.  Improving word-recognizers using an interactive lexicon with active and passive words , 2008, IUI '08.

[19]  Per Ola Kristensson,et al.  Asynchronous Multimodal Text Entry Using Speech and Gesture Keyboards , 2011, INTERSPEECH.

[20]  K. M. Ponting,et al.  Computational Models of Speech Pattern Processing , 1999, NATO ASI Series.

[21]  Joshua Goodman,et al.  Language modeling for soft keyboards , 2002, IUI '02.

[22]  Tim Paek,et al.  Sampling representative phrase sets for text entry experiments: a procedure and public resource , 2011, CHI.

[23]  Per Ola Kristensson,et al.  Uncertain text entry on mobile devices , 2014, CHI.

[24]  Nils Klarlund,et al.  Word N-Grams for Cluster Keyboards , 2003 .

[25]  Anthony J. Robinson,et al.  Language model adaptation using mixtures and an exponentially decaying cache , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[26]  Kumiko Tanaka-Ishii.  Word-based predictive text entry using adaptive language models , 2007, Nat. Lang. Eng..

[27]  Daniel J. Wigdor,et al.  Typing on flat glass: examining ten-finger expert typing patterns on touch surfaces , 2011, CHI.

[28]  Stanley F. Chen,et al.  Evaluation Metrics For Language Models , 1998 .

[29]  Yiming Yang,et al.  Introducing the Enron Corpus , 2004, CEAS.

[30]  Shumin Zhai,et al.  Making touchscreen keyboards adaptive to keys, hand postures, and individuals: a hierarchical spatial backoff model approach , 2013, CHI.

[31]  Shumin Zhai,et al.  Both complete and correct?: multi-objective optimization of touchscreen keyboard , 2014, CHI.

[32]  Shumin Zhai,et al.  Touch behavior with different postures on soft smartphone keyboards , 2012, Mobile HCI.

[33]  Shumin Zhai,et al.  SHARK2: a large vocabulary shorthand writing system for pen-based computers , 2004, UIST '04.

[34]  Renato De Mori,et al.  A Cache-Based Natural Language Model for Speech Recognition , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[35]  Joshua Goodman,et al.  A bit of progress in language modeling , 2001, Comput. Speech Lang..

[36]  Andreas Stolcke,et al.  Entropy-based Pruning of Backoff Language Models , 2000, ArXiv.