Context-aware Stand-alone Neural Spelling Correction

Existing natural language processing systems are vulnerable to noisy inputs resulting from misspellings. On the contrary, humans can easily infer the corresponding correct words from their misspellings and surrounding context. Inspired by this, we address the stand-alone spelling correction problem, which only corrects the spelling of each token without additional token insertion or deletion, by utilizing both spelling information and global context representations. We present a simple yet powerful solution that jointly detects and corrects misspellings as a sequence labeling task by fine-turning a pre-trained language model. Our solution outperforms the previous state-of-the-art result by 12.8% absolute F0.5 score.

[1]  Kevin Duh,et al.  Robsut Wrod Reocginiton via Semi-Character Recurrent Neural Network , 2016, AAAI.

[2]  Michael Flor,et al.  On using context for automatic correction of non-word misspellings in student essays , 2012, BEA@NAACL-HLT.

[3]  Yiming Yang,et al.  XLNet: Generalized Autoregressive Pretraining for Language Understanding , 2019, NeurIPS.

[4]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[5]  Marcos Zampieri,et al.  Effective Spell Checking Methods Using Clustering Algorithms , 2013, RANLP.

[6]  Ben Hutchinson,et al.  Using the Web for Language Independent Spellchecking and Autocorrection , 2009, EMNLP.

[7]  Fred J. Damerau,et al.  A technique for computer detection and correction of spelling errors , 1964, CACM.

[8]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[9]  Gregory V. Bard,et al.  Spelling-Error Tolerant, Order-Independent Pass-Phrases via the Damerau-Levenshtein String-Edit Distance Metric , 2007, ACSW.

[10]  Yang Zhang,et al.  Discriminative Reranking for Spelling Correction , 2006, PACLIC.

[11]  Thorsten Brants,et al.  One billion word benchmark for measuring progress in statistical language modeling , 2013, INTERSPEECH.

[12]  Graeme Hirst,et al.  Real-Word Spelling Correction with Trigrams: A Reconsideration of the Mays, Damerau, and Mercer Model , 2008, CICLing.

[13]  José A. R. Fonollosa,et al.  Dealing with Input Noise in Statistical Machine Translation , 2012, COLING.

[14]  Philip Gage,et al.  A new algorithm for data compression , 1994 .

[15]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[16]  Vladimir I. Levenshtein,et al.  Binary codes capable of correcting deletions, insertions, and reversals , 1965 .

[17]  Jason S. Chang,et al.  機器翻譯為本的中文拼字改錯系統 (Chinese Spelling Checker Based on Statistical Machine Translation) , 2013, ROCLING.

[18]  Ilya Sutskever,et al.  Language Models are Unsupervised Multitask Learners , 2019 .

[19]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[20]  Yonatan Belinkov,et al.  Synthetic and Natural Noise Both Break Neural Machine Translation , 2017, ICLR.

[21]  Ralf Klabunde.  Daniel Jurafsky/James H. Martin, Speech and Language Processing , 2002 .

[22]  Michael Flor,et al.  Four types of context for automatic spelling correction , 2012, TAL.

[23]  George Kurian,et al.  Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation , 2016, ArXiv.

[24]  Xu Sun,et al.  A Large Scale Ranker-Based System for Search Query Spelling Correction , 2010, COLING.

[25]  Eric Brill,et al.  An Improved Error Model for Noisy Channel Spelling Correction , 2000, ACL.

[26]  Andrew Carlson,et al.  Memory-based context-sensitive spelling correction at web scale , 2007, Sixth International Conference on Machine Learning and Applications (ICMLA 2007).

[27]  Yuji Matsumoto,et al.  A Hybrid Chinese Spelling Correction Using Language Model and Statistical Machine Translation with Reranking , 2013, SIGHAN@IJCNLP.

[28]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[29]  Graham Rawlinson,et al.  The Significance of Letter Position in Word Recognition , 2007, IEEE Aerospace and Electronic Systems Magazine.

[30]  Robert L. Mercer,et al.  Context based spelling correction , 1991, Inf. Process. Manag..

[31]  Eric Brill,et al.  Spelling Correction as an Iterative Process that Exploits the Collective Knowledge of Web Users , 2004, EMNLP.

[32]  Zhiwei Wang,et al.  Learning Multi-level Dependencies for Robust Word Recognition , 2019, AAAI.

[33]  Harshit Pande.  Effective search space reduction for spell correction using character neural embeddings , 2017, EACL.

[34]  Luke S. Zettlemoyer,et al.  Deep Contextualized Word Representations , 2018, NAACL.

[35]  Yang Wang,et al.  Spelling Error Correction Using a Nested RNN Model and Pseudo Training Data , 2018, ArXiv.

[36]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[37]  Houfeng Wang,et al.  A Unified Framework for Grammar Error Correction , 2014, CoNLL Shared Task.

[38]  Marcin Junczys-Dowmunt,et al.  Approaching Neural Grammatical Error Correction as a Low-Resource Machine Translation Task , 2018, NAACL.

[39]  Karen Kukich,et al.  Techniques for automatically correcting words in text , 1992, CSUR.

[40]  Hao Tian,et al.  ERNIE 2.0: A Continual Pre-training Framework for Language Understanding , 2019, AAAI.

[41]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[42]  Manoj Kumar Chinnakotla,et al.  Automatic Spelling Correction for Resource-Scarce Languages using Deep Learning , 2018, ACL.

[43]  Saab Mansour,et al.  Spelling Correction of User Search Queries through Statistical Machine Translation , 2015, EMNLP.

[44]  Erik F. Tjong Kim Sang,et al.  Representing Text Chunks , 1999, EACL.