Integrated Chinese Word Segmentation in Statistical Machine Translation

Integrated Chinese Word Segmentation in Statistical Machine Translation Jia Xu and Evgeny Matusov and Richard Zens and Hermann Ney Lehrstuhl f¨ur Informatik VI - Computer Science Department RWTH Aachen University D-52056 Aachen, Germany {xujia,matusov,zens,ney}@cs.rwth-aachen.de Abstract A Chinese sentence is represented as a sequence of charac- ters, and words are not separated from each other. In statisti- cal machine translation, the conventional approach is to seg- ment the Chinese character sequence into words during the pre-processing. The training and translation are performed afterwards. However, this method is not optimal for two rea- sons: 1. The segmentations may be erroneous. 2. For a given character sequence, the best segmentation depends on its context and translation. In order to minimize the translation errors, we take dif- ferent segmentation alternatives instead of a single segmen- tation into account and integrate the segmentation process with the search for the best translation. The segmentation de- cision is only taken during the generation of the translation. With this method we are able to translate Chinese text at the character level. The experiments on the IWSLT 2005 task showed improvements in the translation performance using two translation systems: a phrase-based system and a finite state transducer based system. For the phrase-based system, the improvement of the BLEU score is 1.5% absolute. 1. Introduction In Chinese texts, words composed of single or multiple char- acters are not separated by white space, which is different from most of the European languages. In statistical machine translation, the conventional way is to segment the Chinese character sequence into Chinese words before the training and translation. We compared different segmentation methods in [1]. The training and test texts can be segmented into words or used at the character level. In the experiments in [1], the trans- lation results with the previous method outperformed the re- sults with the latter one. Here we continued the investigation on the translation of the text at the character level and developed a new method yielding better translation results than when translation is at the word level. This method handles all the segmentation alternatives in- stead of only the single-best segmentation. The single-best one may contain errors or may be not optimal with respect to the training corpus. Instead of reading a single best segmented sentence, our system handles all the segmentation alternatives by reading a segmentation lattice. Similar approaches were applied in the speech translation, e.g. [2], where the speech recognition and text translation are combined by using the recognition lattices. We also weight the different segmentations with a language model trained on the Chinese corpus at the word level. Weighting the word segmentation by language model cost was introduced in [3]. To verify the improvements with the integrated segmen- tation method, we experimented on two translation systems: translation with the weighted finite state transducers and translation with the phrase based approach. On the IWSLT 2005 task [4], using a phrase-based translation system, the improvement of the BLEU score reached 1.5% absolute. This paper is structured as …

Content truncated to 500 words