Oracle-based Training for Phrase-based Statistical Machine Translation

Oracle-based Training for Phrase-based Statistical Machine Translation Ankit K. Srivastava CNGL, School of Computing Dublin City University, Ireland asrivastava@computing.dcu.ie Yanjun Ma Baidu Inc. Beijing, China yma@baidu.com Andy Way CNGL, School of Computing Dublin City University, Ireland away@computing.dcu.ie Abstract A Statistical Machine Translation (SMT) system generates an n-best list of candidate translations for each sentence. A model er- ror occurs if the most probable translation (1-best) generated by the SMT decoder is not the most accurate as measured by its similarity to the human reference transla- tion(s) (an oracle). In this paper we inves- tigate the parametric differences between the 1-best and the oracle translation and at- tempt to try and close this gap by propos- ing two rescoring strategies to push the or- acle up the n-best list. We observe modest improvements in METEOR scores over the baseline SMT system trained on French– English Europarl corpora. We present a de- tailed analysis of the oracle rankings to de- termine the source of model errors, which in turn has the potential to improve overall system performance. 1 Introduction Phrase-based Statistical Machine Translation (PB- SMT) systems typically learn translation, reorder- ing, and target-language features from a large number of parallel sentences. Such features are then combined in a log-linear model (Och and Ney, 2002), the coefficients of which are optimized on an objective function measuring translation quality such as the BLEU metric (Papineni et al., 2002), using Minimum Error Rate Training (MERT) as described in Och (2003). An SMT decoder non-exhaustively explores the exponential search space of translations for each source sentence, scoring each hypothesis using the c⃝ 2011 European Association for Machine Translation. formula (Och and Ney, 2002) in (1). P(e|f) = exp( M � i=1 λihi(e, f)) (1) The variable h denotes each of the M fea- tures (probabilities learned from language models, translation models, etc.) and λ denotes the associ- ated feature weight (coefficient). The candidate translation (in the n-best list) hav- ing the highest decoder score is deemed to be the best translation (1-best) according to the model. Automatic evaluation metrics measuring similarity to human reference translations can be modified to generate a score on the sentence level instead of at system level. These scores can, in turn, be used to determine the quality or goodness of a transla- tion. The candidate having the highest sentence- level evaluation score is deemed to be the most ac- curate translation (oracle). In practice, it has been found that the n-best list rankings can be fairly poor (i.e. low proportion of oracles in rank 1), and the oracle translations (the candidates closest to a reference translation as measured by automatic evaluation metrics) occur much lower in the list. Model errors (Germann et al., 2004) occur when the optimum translation (1- best) is not equivalent to the most accurate transla- tion (oracle). The aim of this paper is to investigate these model errors by quantifying the differences between the 1-best and the oracle translations, and evaluate impact of the features used in …

Content truncated to 500 words