Lattice Score Based Data Cleaning for Phrase-Based Statistical Machine Translation

Lattice Score Based Data Cleaning For Phrase-Based Statistical Machine Translation Jie Jiang†, Andy Way†, Julie Carson-Berndsen‡ †CNGL, School of Computing, Dublin City University, Glasnevin, Dublin 9 ‡Department of Computer Science, University College Dublin, Belfield, Dublin 4 {jjiang,away}@computing.dcu.ie, julie.berndsen@ucd.ie Abstract Statistical machine translation relies heav- ily on parallel corpora to train its mod- els for translation tasks. While more and more bilingual corpora are readily avail- able, the quality of the sentence pairs should be taken into consideration. This paper presents a novel lattice score-based data cleaning method to select proper sen- tence pairs from the ones extracted from a bilingual corpus by the sentence alignment methods. The proposed method is carried out as follows: firstly, an initial phrase- based model is trained on the full sentence- aligned corpus; then for each of the sen- tence pairs in the corpus, word alignments are used to create anchor pairs and source- side lattices; thirdly, based on the trans- lation model, target-side phrase networks are expanded on the lattices and Viterbi searching is used to find approximated de- coding results; finally, BLEU score thresh- olds are used to filter out the low-score sentence pairs for the data cleaning pur- pose. Our experiments on the FBIS cor- pus showed improvements of BLEU score from 23.78 to 24.02 in Chinese-English. 1 Introduction To overcome problems of data sparseness, most statistical machine translation (SMT) methods tend to use the largest possible corpora to train the models. Following the word-based (Brown et al., 1993) and phrase-based (Koehn et al., 2003) methods, incorporating as much parallel corpora as c⃝ 2010 European Association for Machine Translation. possible normally obtains a better system perfor- mance. Even in some SMT evaluation tasks (Zoll- mann et al., 2008), a huge amount (nearly 10M) of sentence pairs are provided to train the models. However, as the size of the corpus increases dramatically, sentence alignment can only be per- formed automatically. Since it is inevitable that (a possibly large number of) misaligned sentences will be introduced during the automatic sentence alignment phase, some part of the bilingual corpus will not contribute at all to the SMT systems. An- other drawback of this paradigm is that it takes a long time to train models on the full corpus. Fur- thermore, even if the models are ready for use, sometimes they will become too big to fit into memory, and in addition the decoding speed will suffer. To overcome these problems, two possible methods may be useful for improving conven- tional SMT methods: (i) phrase table pruning tech- niques (Johnson et al., 2007; Yang et al., 2009), which are carried out on the model side; and (ii) data cleaning methods, which are carried out ab initio on the training data side. Some work has been reported on data cleaning of SMT. In (L¨u et al., 2007), information retrieval methods are utilized to weight the training data to obtain significant improvements over a base- line SMT system. In (Okita, 2009), both word- based and phrase-based models are …

Content truncated to 500 words