Toward a unified approach to lexicon optimization and perplexity minimization for Chinese language modeling

This paper presents a unified approach to lexicon optimization and perplexity minimization for Chinese language modeling (LM). Instead of using a non-iterative segmentation-detection method, the proposed approach iteratively extracts candidate words, selects new words based on a perplexity minimization criterion and adds the new words into the lexicon. The augmented lexicon, which contains the new words, is used in the next iteration to re-segment the input corpus until the perplexity of the LM is converged. The experiments show that both the precision and recall rates are improved and the perplexity of the LM has reduced 6.3%.