Unsupervised Tokenization for Machine Translation

Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 718–726, Singapore, 6-7 August 2009. c⃝2009 ACL and AFNLP Unsupervised Tokenization for Machine Translation Tagyoung Chung and Daniel Gildea Computer Science Department University of Rochester Rochester, NY 14627 Abstract Training a statistical machine translation starts with tokenizing a parallel corpus. Some languages such as Chinese do not in- corporate spacing in their writing system, which creates a challenge for tokenization. Moreover, morphologically rich languages such as Korean present an even bigger challenge, since optimal token boundaries for machine translation in these languages are often unclear. Both rule-based solu- tions and statistical solutions are currently used. In this paper, we present unsuper- vised methods to solve tokenization prob- lem. Our methods incorporate informa- tion available from parallel corpus to de- termine a good tokenization for machine translation. 1 Introduction Tokenizing a parallel corpus is usually the first step of training a statistical machine translation system. With languages such as Chinese, which has no spaces in its writing system, the main chal- lenge is to segment sentences into appropriate to- kens. With languages such as Korean and Hun- garian, although the writing systems of both lan- guages incorporate spaces between “words”, the granularity is too coarse compared with languages such as English. A single word in these lan- guages is composed of several morphemes, which often correspond to separate words in English. These languages also form compound nouns more freely. Ideally, we want to find segmentations for source and target languages that create a one-to- one mapping of words. However, this is not al- ways straightforward for two major reasons. First, what the optimal tokenization for machine trans- lation should be is not always clear. Zhang et al. (2008b) and Chang et al. (2008) show that get- ting the tokenization of one of the languages in the corpus close to a gold standard does not nec- essarily help with building better machine trans- lation systems. Second, even statistical methods require hand-annotated training data, which means that in resource-poor languages, good tokenization is hard to achieve. In this paper, we explore unsupervised methods for tokenization, with the goal of automatically finding an appropriate tokenization for machine translation. We compare methods that have ac- cess to parallel corpora to methods that are trained solely using data from the source language. Unsu- pervised monolingual segmentation has been stud- ied as a model of language acquisition (Goldwater et al., 2006), and as model of learning morphol- ogy in European languages (Goldsmith, 2001). Unsupervised segmentation using bilingual data has been attempted for finding new translation pairs (Kikui and Yamamoto, 2002), and for finding good segmentation for Chinese in machine trans- lation using Gibbs sampling (Xu et al., 2008). In this paper, further investigate the use of bilingual information to find tokenizations tailored for ma- chine translation. We find a benefit not only for segmentation of languages with no space in the writing system (such as Chinese), but also for the smaller-scale tokenization problem of …

Content truncated to 500 words