Bayesian Semi-Supervised Chinese Word Segmentation for Statistical Machine Translation

Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 1017–1024 Manchester, August 2008 Bayesian Semi-Supervised Chinese Word Segmentation for Statistical Machine Translation Jia Xu†, Jianfeng Gao∗, Kristina Toutanova∗, Hermann Ney† Computer Science 6† Microsoft Corporation∗ RWTH Aachen University One Microsoft Way D-52056 Aachen, Germany Redmond, WA 98052, USA {xujia,ney}@cs.rwth-aachen.de {jfgao,kristout}@microsoft.com Abstract Words in Chinese text are not naturally separated by delimiters, which poses a challenge to standard machine translation (MT) systems. In MT, the widely used approach is to apply a Chinese word seg- menter trained from manually annotated data, using a fixed lexicon. Such word segmentation is not necessarily optimal for translation. We propose a Bayesian semi-supervised Chinese word segmenta- tion model which uses both monolingual and bilingual information to derive a seg- mentation suitable for MT. Experiments show that our method improves a state-of- the-art MT system in a small and a large data environment. 1 Introduction Chinese sentences are written in the form of a se- quence of Chinese characters, and words are not separated by white spaces. This is different from most European languages and poses difficulty in many natural language processing tasks, such as machine translation. It is difficult to define “correct” Chinese word segmentation (CWS) and various definitions have been proposed. In this work, we explore the idea that the best segmentation depends on the task, and concentrate on developing a CWS method for MT, which leads to better translation performance. The common solution in Chinese-to-English translation has been to segment the Chinese text using an off-the-shelf CWS method, and to apply a standard translation model given the fixed seg- mentation. The most widely applied method for MT is unigram segmentation, such as segmenta- tion using the LDC (LDC, 2003) tool, which re- quires a manual lexicon containing a list of Chi- nese words and their frequencies. The lexicon and c⃝ 2008. Licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported li- cense (http://creativecommons.org/licenses/by-nc-sa/3.0/). Some rights reserved. frequencies are obtained using manually annotated data. This method is sub-optimal for MT. For ex- ample, �(paper) and �(card) can be two words or composed into one word ��(cards). Since � �does not exist in the manual lexicon, it cannot be generated by this method. In addition to unigram segmentation, other methods have been proposed. For example, (Gao et al., 2005) described an adaptive CWS system, and (Andrew, 2006) employed a conditional ran- dom field model for sequence segmentation. How- ever, these methods are not specifically devel- oped for the MT application, and significant im- provements in translation performance need to be shown. In (Xu et al., 2004) and (Xu et al., 2005), word segmentations are integrated into MT sys- tems during model training and translation. We re- fine the method in training using a Bayesian semi- supervised CWS approach motivated by (Goldwa- ter et al., 2006). We describe a generative model which consists of a word model and two alignment models, representing the monolingual and bilin- gual information, respectively. In our methods, we …

Content truncated to 500 words