Joint Tokenization and Translation

Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 1200–1208, Beijing, August 2010 Joint Tokenization and Translation Xinyan Xiao † Yang Liu † Young-Sook Hwang ‡ Qun Liu † Shouxun Lin † †Key Lab. of Intelligent Info. Processing ‡HILab Convergence Technology Center Institute of Computing Technology C&I Business Chinese Academy of Sciences SKTelecom {xiaoxinyan,yliu,liuqun,sxlin}@ict.ac.cn yshwang@sktelecom.com Abstract As tokenization is usually ambiguous for many natural languages such as Chinese and Korean, tokenization errors might po- tentially introduce translation mistakes for translation systems that rely on 1-best to- kenizations. While using lattices to of- fer more alternatives to translation sys- tems have elegantly alleviated this prob- lem, we take a further step to tokenize and translate jointly. Taking a sequence of atomic units that can be combined to form words in different ways as input, our joint decoder produces a tokenization on the source side and a translation on the target side simultaneously. By integrat- ing tokenization and translation features in a discriminative framework, our joint decoder outperforms the baseline trans- lation systems using 1-best tokenizations and lattices significantly on both Chinese- English and Korean-Chinese tasks. In- terestingly, as a tokenizer, our joint de- coder achieves significant improvements over monolingual Chinese tokenizers. 1 Introduction Tokenization plays an important role in statistical machine translation (SMT) because tokenizing a source-language sentence is always the first step in SMT systems. Based on the type of input, Mi and Huang (2008) distinguish between two cat- egories of SMT systems : string-based systems (Koehn et al., 2003; Chiang, 2007; Galley et al., source target tokenize+translate string tokenization translation source target string tokenize tokenization translate translation (a) (b) Figure 1: (a) Separate tokenization and translation and (b) joint tokenization and translation. 2006; Shen et al., 2008) that take a string as input and tree-based systems (Liu et al., 2006; Mi et al., 2008) that take a tree as input. Note that a tree- based system still needs to first tokenize the input sentence and then obtain a parse tree or forest of the sentence. As shown in Figure 1(a), we refer to this pipeline as separate tokenization and transla- tion because they are divided into single steps. As tokenization for many languages is usually ambiguous, SMT systems that separate tokeniza- tion and translation suffer from a major drawback: tokenization errors potentially introduce transla- tion mistakes. As some languages such as Chi- nese have no spaces in their writing systems, how to segment sentences into appropriate words has a direct impact on translation performance (Xu et al., 2005; Chang et al., 2008; Zhang et al., 2008). In addition, although agglutinative languages such as Korean incorporate spaces between “words”, which consist of multiple morphemes, the gran- ularity is too coarse and makes the training data 1200 considerably sparse. Studies reveal that seg- menting “words” into morphemes effectively im- proves translating morphologically rich languages (Oflazer, 2008). More importantly, a tokenization close to a gold standard does not necessarily leads to better translation quality (Chang et al., 2008; Zhang …

Content truncated to 500 words