Neural Network Language Model for Chinese Pinyin Input Method Engine

Neural network language models (NNLMs) have been shown to outperform traditional ngram language model. However, too high computational cost of NNLMs becomes the main obstacle of directly integrating it into pinyin IME that normally requires a real-time response. In this paper, an efficient solution is proposed by converting NNLMs into back-off n-gram language models, and we integrate the converted NNLM into pinyin IME. Our experimental results show that the proposed method gives better decoding predictive performance for pinyin IME with satisfied efficiency.

[1]  Holger Schwenk,et al.  Continuous Space Language Models for Statistical Machine Translation , 2006, ACL.

[2]  Holger Schwenk,et al.  CSLM - a modular open-source continuous space language modeling toolkit , 2013, INTERSPEECH.

[3]  Ashish Vaswani,et al.  Decoding with Large-Scale Neural Language Models Improves Translation , 2013, EMNLP.

[4]  Hai Zhao,et al.  A Joint Graph Model for Pinyin-to-Chinese Conversion with Typo Correction , 2014, ACL.

[5]  Zheng Chen,et al.  A New Statistical Approach To Chinese Pinyin Input , 2000, ACL.

[6]  Hai Zhao,et al.  A Machine Learning Approach to Convert CCGbank to Penn Treebank , 2012, COLING.

[7]  F ChenStanley,et al.  An Empirical Study of Smoothing Techniques for Language Modeling , 1996, ACL.

[8]  Hai Zhao,et al.  A New Word Language Model Evaluation Metric for Character Based Languages , 2013, CCL.

[9]  Hai Zhao,et al.  A Machine Translation Approach for Chinese Whole-Sentence Pinyin-to-Character Conversion , 2012 .

[10]  Hai Zhao,et al.  Converting Continuous-Space Language Models into N-Gram Language Models for Statistical Machine Translation , 2013, EMNLP.

[11]  Yoshua Bengio,et al.  A Neural Probabilistic Language Model , 2003, J. Mach. Learn. Res..

[12]  Andreas Stolcke,et al.  SRILM - an extensible language modeling toolkit , 2002, INTERSPEECH.

[13]  Yuji Matsumoto,et al.  Chinese Word Segmentation by Classification of Characters , 2005, Int. J. Comput. Linguistics Chin. Lang. Process..

[14]  Ebru Arisoy,et al.  Converting Neural Network Language Models into Back-off Language Models for Efficient Decoding in Automatic Speech Recognition , 2013, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[15]  Jr. G. Forney,et al.  Viterbi Algorithm , 1973, Encyclopedia of Machine Learning.

[16]  Spyridon Matsoukas,et al.  BBN's Systems for the Chinese-English Sub-task of the NTCIR-10 PatentMT Evaluation , 2013, NTCIR.

[17]  Hai Zhao,et al.  An Empirical Study on Word Segmentation for Chinese Machine Translation , 2013, CICLing.

[18]  Hai Zhao,et al.  Neural Network Based Bilingual Language Model Growing for Statistical Machine Translation , 2014, EMNLP.

[19]  Andrew J. Viterbi,et al.  Error bounds for convolutional codes and an asymptotically optimum decoding algorithm , 1967, IEEE Trans. Inf. Theory.

[20]  Hai Zhao,et al.  Learning Hierarchical Translation Spans , 2014, EMNLP.

[21]  Richard M. Schwartz,et al.  Fast and Robust Neural Network Joint Models for Statistical Machine Translation , 2014, ACL.

[22]  Alexandre Allauzen,et al.  Training Continuous Space Language Models: Some Practical Issues , 2010, EMNLP.

[23]  Hai Zhao,et al.  An Improved Chinese Word Segmentation System with Conditional Random Field , 2006, SIGHAN@COLING/ACL.

[24]  Hai Zhao,et al.  Using Deep Linguistic Features for Finding Deceptive Opinion Spam , 2012, COLING.

[25]  Holger Schwenk,et al.  Continuous space language models , 2007, Comput. Speech Lang..

[26]  Hai Zhao,et al.  Bilingual Continuous-Space Language Model Growing for Statistical Machine Translation , 2015, IEEE/ACM Transactions on Audio, Speech, and Language Processing.