Optimizing Chinese Word Segmentation for Machine Translation Performance

Proceedings of the Third Workshop on Statistical Machine Translation, pages 224–232, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Optimizing Chinese Word Segmentation for Machine Translation Performance Pi-Chuan Chang, Michel Galley, and Christopher D. Manning Computer Science Department, Stanford University Stanford, CA 94305 pichuan,galley,manning@cs.stanford.edu Abstract Previous work has shown that Chinese word seg- mentation is useful for machine translation to En- glish, yet the way different segmentation strategies affect MT is still poorly understood. In this pa- per, we demonstrate that optimizing segmentation for an existing segmentation standard does not al- ways yield better MT performance. We find that other factors such as segmentation consistency and granularity of Chinese “words” can be more impor- tant for machine translation. Based on these find- ings, we implement methods inside a conditional random field segmenter that directly optimize seg- mentation granularity with respect to the MT task, providing an improvement of 0.73 BLEU. We also show that improving segmentation consistency us- ing external lexicon and proper noun features yields a 0.32 BLEU increase. 1 Introduction Word segmentation is considered an important first step for Chinese natural language processing tasks, because Chinese words can be composed of multi- ple characters but with no space appearing between words. Almost all tasks could be expected to ben- efit by treating the character sequence “Us” to- gether, with the meaning smallpox, rather than deal- ing with the individual characters “U” (sky) and “s” (flower). Without a standardized notion of a word, traditionally, the task of Chinese word seg- mentation starts from designing a segmentation stan- dard based on linguistic and task intuitions, and then aiming to building segmenters that output words that conform to the standard. One widely used standard is the Penn Chinese Treebank (CTB) Segmentation Standard (Xue et al., 2005). It has been recognized that different NLP ap- plications have different needs for segmentation. Chinese information retrieval (IR) systems benefit from a segmentation that breaks compound words into shorter “words” (Peng et al., 2002), parallel- ing the IR gains from compound splitting in lan- guages like German (Hollink et al., 2004), whereas automatic speech recognition (ASR) systems prefer having longer words in the speech lexicon (Gao et al., 2005). However, despite a decade of very in- tense work on Chinese to English machine transla- tion (MT), the way in which Chinese word segmen- tation affects MT performance is very poorly under- stood. With current statistical phrase-based MT sys- tems, one might hypothesize that segmenting into small chunks, including perhaps even working with individual characters would be optimal. This is be- cause the role of a phrase table is to build domain and application appropriate larger chunks that are semantically coherent in the translation process. For example, even if the word for smallpox is treated as two one-character words, they can still appear in a phrase like “U s→smallpox”, so that smallpox will still be a candidate translation when the system translates “U” “s”. Nevertheless, Xu et al. (2004) show that an MT system …

Content truncated to 500 words