Improved Statistical Machine Translation by Multiple Chinese Word Segmentation

Proceedings of the Third Workshop on Statistical Machine Translation, pages 216–223, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Improved Statistical Machine Translation by Multiple Chinese Word Segmentation Ruiqiang Zhang1,2 and Keiji Yasuda1,2 and Eiichiro Sumita1,2 1National Institute of Information and Communications Technology 2ATR Spoken Language Communication Research Laboratories 2-2-2 Hikaridai, Science City, Kyoto, 619-0288, Japan {ruiqiang.zhang,keiji.yasuda,eiichiro.sumita}@atr.jp Abstract Chinese word segmentation (CWS) is a necessary step in Chinese-English statisti- cal machine translation (SMT) and its per- formance has an impact on the results of SMT. However, there are many settings in- volved in creating a CWS system such as various specifications and CWS methods. This paper investigates the effect of these settings to SMT. We tested dictionary- based and CRF-based approaches and found there was no significant difference between the two in the qualty of the result- ing translations. We also found the corre- lation between the CWS F-score and SMT BLEU score was very weak. This paper also proposes two methods of combining advantages of different specifications: a simple concatenation of training data and a feature interpolation approach in which the same types of features of translation models from various CWS schemes are linearly interpolated. We found these ap- proaches were very effective in improving quality of translations. 1 Introduction Chinese word segmentation (CWS) is a necessary step in Chinese-English statistical machine transla- tion (SMT). The research on CWS independently from SMT has been conducted for decades. As an evidence, the CWS evaluation campaign, the Sighan Bakeoff (Emerson, 2005),1, has been held four times since 2004. However, works on relations between CWS and SMT are scarce. Generally, two factors need to be considered in constructing a CWS system. The first one is the specifications for CWS, i.e., the rules or guidelines for word segmentation, and the second one is the CWS methods. There are many CWS specifications used by different organizations. Unfortunately, these organizations do not seem to have any intention of reaching a unified specification. More than five or six specifications have been used in the four Sighan Bakeoffs. There is also significant disagreement on the specifications, although much of their contents is the same. One of the aims of this work was therefore to establish whether inconsistencies in specifications significantly affect the quality of SMT. The second factor is CWS methods. We grouped all of the CWS methods into two classes: the class without out-of-vocabulary (OOV) recognition and the class with OOV recognition, represented by the dictionary-based CWS and the CRF-based CWS, re- spectively. Out-of-vocabulary recognition may have two-sided effects on SMT performance. The CRF- based CWS that supports OOV recognition produces word segmentations with a higher F-score, but a huge number of new words recognized correctly and incorrectly that can incur data sparseness in training the SMT models. On the other hand, the dictionary- based approach that does not support OOV recogni- tion produced a lower F-score, but with a relatively weak data spareness problem. Which approach pro- 1A CWS competition organized by the …

Content truncated to 500 words