Marker-Based Filtering of Bilingual Phrase Pairs for SMT

Proceedings of the 13th Annual Conference of the EAMT, pages 144–151, Barcelona, May 2009 Marker-based Filtering of Bilingual Phrase Pairs for SMT Felipe S´anchez-Mart´ınez Dept. Llenguatges i Sistemes Inform`atics Universitat d’Alacant E-03071 Alacant, Spain fsanchez@dlsi.ua.es Andy Way NCLT, School of Computing Dublin City University Dublin 9, Ireland away@computing.dcu.ie Abstract State-of-the-art statistical machine transla- tion systems make use of a large trans- lation table obtained after scoring a set of bilingual phrase pairs automatically ex- tracted from a parallel corpus. The num- ber of bilingual phrase pairs extracted from a pair of aligned sentences grows expo- nentially as the length of the sentences in- creases; therefore, the number of entries in the phrase table used to carry out the translation may become unmanageable, es- pecially when online, ‘on demand’ trans- lation is required in real time. We de- scribe the use of closed-class words to fil- ter the set of bilingual phrase pairs ex- tracted from the parallel corpus by tak- ing into account the alignment information and the type of the words involved in the alignments. On four European language pairs, we show that our simple yet novel approach can filter the phrase table by up to a third yet still provide competitive results compared to the baseline. Furthermore, it provides a nice balance between the un- filtered approach and pruning using stop words, where the deterioration in transla- tion quality is unacceptably high. 1 Introduction The state-of-the-art statistical approach to ma- chine translation (MT) is the phrase-based model. Phrase-based statistical MT (PB-SMT) systems (Zens et al., 2002; Koehn et al., 2003) are based on the log linear model combination of sev- eral feature functions (Och and Ney, 2002), one c⃝ 2009 European Association for Machine Translation. of which is the phrase translation probability esti- mated after extracting bilingual phrase pairs from the parallel corpus.1 Bilingual phrase pairs are automatically extracted after computing the word align- ments (Brown et al., 1993; Och and Ney, 2003). The set BP(sJ 1 , tI 1, A) of bilingual phrase pairs extracted from the word-aligned sentence pair sI 1 = (s1, . . . , si, . . . , sI) and tJ 1 = (t1, . . . , tj, . . . , tJ) is defined as in (1) (Zens et al., 2002): (1) BP(sI 1, tJ 1 , A) = {(si+n i , tj+m j ) : ∀(i′, j′) ∈ A : i ≤ i′ ≤ i + n ⇔ j ≤ j′ ≤ j + m}, where A = {(i, j) : i ∈ [1, I] ∧ j ∈ [1, J]} is a set of pairs with the alignment information between the words in the source sentence sI 1 and the words in the target sentence tJ 1 . According to equation (1), all words within a bilingual phrase pair are consecutive and not aligned with words from outside the bilingual phrase pair. It is worth noting that bilingual phrase pairs may contain words that are not aligned at all, …

Content truncated to 500 words