Integration of Multiple Bilingually-Learned Segmentation Schemes into Statistical Machine Translation

Proceedings of the Joint 5th Workshop on Statistical Machine Translation and MetricsMATR, pages 400–408, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics Integration of Multiple Bilingually-Learned Segmentation Schemes into Statistical Machine Translation Michael Paul and Andrew Finch and Eiichiro Sumita MASTAR Project National Institute of Information and Communications Technology Hikaridai 2-2-2, Keihanna Science City 619-0288 Kyoto, Japan michael.paul@nict.go.jp Abstract This paper proposes an unsupervised word segmentation algorithm that identi- fies word boundaries in continuous source language text in order to improve the translation quality of statistical machine translation (SMT) approaches. The method can be applied to any language pair where the source language is unseg- mented and the target language segmen- tation is known. First, an iterative boot- strap method is applied to learn multi- ple segmentation schemes that are consis- tent with the phrasal segmentations of an SMT system trained on the resegmented bitext. In the second step, multiple seg- mentation schemes are integrated into a single SMT system by characterizing the source language side and merging iden- tical translation pairs of differently seg- mented SMT models. Experimental re- sults translating five Asian languages into English revealed that the method of in- tegrating multiple segmentation schemes outperforms SMT models trained on any of the learned word segmentations and performs comparably to available state-of- the-art monolingually-built segmentation tools. 1 Introduction The task of word segmentation, i.e., identifying word boundaries in continuous text, is one of the fundamental preprocessing steps of data-driven NLP applications like Machine Translation (MT). In contrast to Indo-European languages like En- glish, many Asian languages like Chinese do not use a whitespace character to separate meaningful word units. The problems of word segmentation are: (1) ambiguity, e.g., for Chinese, a single charac- ter can be a word component in one context, but a word by itself in another context. (2) unknown words, i.e., existing words can be combined into new words such as proper nouns, e.g. “White House”. Purely dictionary-based approaches like (Cheng et al., 1999) addressed these problems by max- imum matching heuristics. Recent research on unsupervised word segmentation focuses on ap- proaches based on probabilistic methods. For ex- ample, (Brent, 1999) proposes a probabilistic seg- mentation model based on unigram word distri- butions, whereas (Venkataraman, 2001) uses stan- dard n-gram language models. An alternative non- parametric Bayesian inference approach based on the Dirichlet process incorporating unigram and bigram word dependencies is introduced in (Gold- water et al., 2006). The focus of this paper, however, is to learn word segmentations that are consistent with phrasal segmentations of SMT translation mod- els. In case of small translation units, e.g. sin- gle Chinese or Japanese characters, it is likely that such tokens have been seen in the training corpus, thus these tokens can be translated by an SMT engine. However, the contextual infor- mation provided by these tokens might not be enough to obtain a good translation. For exam- ple, a Japanese-English SMT engine might trans- late the two successive characters “ � ” (“white”) and …

Content truncated to 500 words