Unsupervised Word Segmentation with Bi-directional Neural Language Model

We present an unsupervised word segmentation model, in which the learning objective is to maximize the generation probability of a sentence given its all possible segmentation. Such generation probability can be factorized into the likelihood of each possible segment given the context in a recursive way. In order to better capture the longand short-term dependencies, we propose to use bi-directional neural language models to better capture the features of segment’s context. Two decoding algorithms are also described to combine the context features from both directions to generate the final segmentation, which helps to reconcile word boundary ambiguities. Experimental results showed that our context-sensitive unsupervised segmentation model achieved state-of-the-art at different evaluation settings on various data sets for Chinese, and the comparable result for Thai.

[1]  Kumiko Tanaka-Ishii,et al.  Unsupervised Segmentation of Chinese Text by Use of Branching Entropy , 2006, ACL.

[2]  Thomas L. Griffiths,et al.  Contextual Dependencies in Unsupervised Word Segmentation , 2006, ACL.

[3]  Benoît Sagot,et al.  Unsupervized Word Segmentation: the Case for Mandarin Chinese , 2012, ACL.

[4]  S. MacEachern,et al.  Estimating mixture of dirichlet process models , 1998 .

[5]  Xiaotie Deng,et al.  Accessor Variety Criteria for Chinese Word Extraction , 2004, CL.

[6]  Naonori Ueda,et al.  Bayesian Unsupervised Word Segmentation with Nested Pitman-Yor Language Modeling , 2009, ACL.

[7]  Hai Zhao,et al.  An Empirical Comparison of Goodness Measures for Unsupervised Chinese Word Segmentation with a Unified Framework , 2008, IJCNLP.

[8]  Baobao Chang,et al.  A Joint Model for Unsupervised Chinese Word Segmentation , 2014, EMNLP.

[9]  J. Pitman,et al.  The two-parameter Poisson-Dirichlet distribution derived from a stable subordinator , 1997 .

[10]  Yoshua Bengio,et al.  A Neural Probabilistic Language Model , 2003, J. Mach. Learn. Res..

[11]  Lukás Burget,et al.  Recurrent neural network based language model , 2010, INTERSPEECH.

[12]  Yonghui Wu,et al.  Exploring the Limits of Language Modeling , 2016, ArXiv.

[13]  Yorick Wilks,et al.  Unsupervised Learning of Word Boundary with Description Length Gain , 1999, CoNLL.

[14]  Ching Y. Suen Computational Studies of the Most Frequent Chinese Words and Sounds , 1986, World Scientific Series in Computer Science.

[15]  Kyle R. Cave,et al.  On the segmentation of Chinese words during reading , 2009, Cognitive Psychology.

[16]  J. M. Cattell THE TIME IT TAKES TO SEE AND NAME OBJECTS , 1886 .

[17]  Qun Liu,et al.  Unsupervised Joint Monolingual Character Alignment and Word Segmentation , 2014, CCL.

[18]  Zhi-Hong Deng,et al.  Unsupervised Neural Word Segmentation for Chinese via Segmental Language Modeling , 2018, EMNLP.

[19]  Zhao Hai,et al.  Chinese Word Segmentation: A Decade Review , 2007 .

[20]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[21]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[22]  Jian Zhu,et al.  A New Unsupervised Approach to Word Segmentation , 2011, CL.

[23]  Chong Wang,et al.  Sequence Modeling via Segmentations , 2017, ICML.

[24]  Yee Whye Teh,et al.  A Hierarchical Bayesian Language Model Based On Pitman-Yor Processes , 2006, ACL.

[25]  Yee Whye Teh,et al.  Sharing Clusters among Related Groups: Hierarchical Dirichlet Processes , 2004, NIPS.

[26]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[27]  Noah A. Smith,et al.  Segmental Recurrent Neural Networks , 2015, ICLR.

[28]  R. Sproat A statistical method for finding word boundaries in Chinese text , 1990 .

[29]  Daichi Mochihashi,et al.  Inducing Word and Part-of-Speech with Pitman-Yor Hidden Semi-Markov Models , 2015, ACL.

[30]  T. Griffiths,et al.  A Bayesian framework for word segmentation: Exploring the effects of context , 2009, Cognition.

[31]  Thomas Emerson,et al.  The Second International Chinese Word Segmentation Bakeoff , 2005, IJCNLP.