Discriminative Pruning of Language Models for Chinese Word Segmentation

Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 1001–1008, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Discriminative Pruning of Language Models for Chinese Word Segmentation Jianfeng Li Haifeng Wang Dengjun Ren Guohua Li Toshiba (China) Research and Development Center 5/F., Tower W2, Oriental Plaza, No.1, East Chang An Ave., Dong Cheng District Beijing, 100738, China {lijianfeng, wanghaifeng, rendengjun, liguohua}@rdc.toshiba.com.cn Abstract This paper presents a discriminative pruning method of n-gram language model for Chinese word segmentation. To reduce the size of the language model that is used in a Chinese word segmenta- tion system, importance of each bigram is computed in terms of discriminative pruning criterion that is related to the per- formance loss caused by pruning the bi- gram. Then we propose a step-by-step growing algorithm to build the language model of desired size. Experimental re- sults show that the discriminative pruning method leads to a much smaller model compared with the model pruned using the state-of-the-art method. At the same Chinese word segmentation F-measure, the number of bigrams in the model can be reduced by up to 90%. Correlation be- tween language model perplexity and word segmentation performance is also discussed. 1 Introduction Chinese word segmentation is the initial stage of many Chinese language processing tasks, and has received a lot of attention in the literature (Sproat et al., 1996; Sun and Tsou, 2001; Zhang et al., 2003; Peng et al., 2004). In Gao et al. (2003), an approach based on source-channel model for Chinese word segmentation was pro- posed. Gao et al. (2005) further developed it to a linear mixture model. In these statistical models, language models are essential for word segmen- tation disambiguation. However, an uncom- pressed language model is usually too large for practical use since all realistic applications have memory constraints. Therefore, language model pruning techniques are used to produce smaller models. Pruning a language model is to eliminate a number of parameters explicitly stored in it, according to some pruning criteria. The goal of research for language model pruning is to find criteria or methods, using which the model size could be reduced effectively, while the perform- ance loss is kept as small as possible. A few criteria have been presented for lan- guage model pruning, including count cut-off (Jelinek, 1990), weighted difference factor (Seymore and Rosenfeld, 1996), Kullback- Leibler distance (Stolcke, 1998), rank and en- tropy (Gao and Zhang, 2002). These criteria are general for language model pruning, and are not optimized according to the performance of lan- guage model in specific tasks. In recent years, discriminative training has been introduced to natural language processing applications such as parsing (Collins, 2000), ma- chine translation (Och and Ney, 2002) and lan- guage model building (Kuo et al., 2002; Roark et al., 2004). To the best of our knowledge, it has not been applied to language model pruning. In this paper, we propose a discriminative pruning method of n-gram language model for Chinese word segmentation. …

Content truncated to 500 words