Sentence segmentation using IBM word alignment model 1

Sentence Segmentation Using IBM Word Alignment Model 1 Jia Xu and Richard Zens and Hermann Ney Chair of Computer Science VI, Computer Science Department RWTH Aachen University D-52056 Aachen, Germany {xujia,zens,ney}@cs.rwth-aachen.de Abstract. In statistical machine translation, word alignment models are trained on bilingual corpora. Long sentences pose severe problems: 1. the high computational requirements; 2. the poor quality of the resulting word alignment. We present a sentence-segmentation method that solves these problems by splitting long sentence pairs. Our approach uses the lexicon information to locate the optimal split point. This method is evaluated on two Chinese-English translation tasks in the news domain. We show that the segmentation of long sentences before training significantly improves the final translation quality of a state-of-the-art machine translation system. In one of the tasks, we achieve an improvement of the BLEU score of more than 20% relative. 1 Introduction 1.1 Problem Description In a statistical machine translation system, we de- fine a mathematical model, train the model param- eters on the parallel sentence-aligned corpora and translate the test text with this model and its param- eters. In practice, many sentences in the training cor- pora are long. Some translation applications can- not handle a sentence whose length is larger than a predetermined value. The reasons are memory limits and the computational complexity of the al- gorithms. Therefore, long sentences are usually re- moved during the preprocessing. But even if long sentences are included, the resulting quality is usu- ally not as good as it is for short sentences. 1.2 Comparison with Sentence Alignment The problem of sentence segmentation is similar to the problem of sentence alignment which was investigated by (Brown et al., 1991; Chen, 1993; Moore, 2002). In the case of the sentence segmen- tation, we assume that the sentence pairs are aligned correctly. The tasks are to find appropriate split points and to align the subsentences. In the case of the sentence alignment, the corpus is aligned at the document level only. Here, we have to align the sentences of two documents rather than having to find appropriate split points. 1.3 State of the Art Previous research on the sentence segmentation problem can be found in (Nevado et al., 2003), who searches for the segmentation boundaries using a dynamic programming algorithm. This technique is based on the lexicon information. However, it only allows a monotone alignment of the bilingual seg- mented sentences and it requires a list of manually defined anchor words. 1.4 Idea of the Method Inspired by the phrase extraction approach (Vogel et al., 2004), we introduce a new sentence segmen- tation method which does not need anchor words and allows for nonmonotone alignments of the sub- sentences. Here we separate a sentence pair into two sub- pairs with the so-called “IBM Word Alignment Model 1”. This process is done recursively over all the sub-sentences until their lengths are smaller than a given value. This simple algorithm leads to a significant improvement in translation quality and a speed-up of the training …

Content truncated to 500 words