Learning to Differentiate Better from Worse Translations

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 214–220, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Learning to Differentiate Better from Worse Translations Francisco Guzm´an Shafiq Joty Llu´ıs M`arquez Alessandro Moschitti Preslav Nakov Massimo Nicosia ALT Research Group Qatar Computing Research Institute — Qatar Foundation {fguzman,sjoty,lmarquez,amoschitti,pnakov,mnicosia}@qf.org.qa Abstract We present a pairwise learning-to-rank approach to machine translation evalua- tion that learns to differentiate better from worse translations in the context of a given reference. We integrate several layers of linguistic information encapsulated in tree-based structures, making use of both the reference and the system output simul- taneously, thus bringing our ranking closer to how humans evaluate translations. Most importantly, instead of deciding upfront which types of features are important, we use the learning framework of preference re-ranking kernels to learn the features au- tomatically. The evaluation results show that learning in the proposed framework yields better correlation with humans than computing the direct similarity over the same type of structures. Also, we show our structural kernel learning (SKL) can be a general framework for MT evaluation, in which syntactic and semantic informa- tion can be naturally incorporated. 1 Introduction We have seen in recent years fast improvement in the overall quality of machine translation (MT) systems. This was only possible because of the use of automatic metrics for MT evaluation, such as BLEU (Papineni et al., 2002), which is the de- facto standard; and more recently: TER (Snover et al., 2006) and METEOR (Lavie and Denkowski, 2009), among other emerging MT evaluation met- rics. These automatic metrics provide fast and in- expensive means to compare the output of differ- ent MT systems, without the need to ask for hu- man judgments each time the MT system has been changed. As a result, this has enabled rapid develop- ment in the field of statistical machine translation (SMT), by allowing to train and tune systems as well as to track progress in a way that highly cor- relates with human judgments. Today, MT evaluation is an active field of re- search, and modern metrics perform analysis at various levels, e.g., lexical (Papineni et al., 2002; Snover et al., 2006), including synonymy and paraphrasing (Lavie and Denkowski, 2009); syn- tactic (Gim´enez and M`arquez, 2007; Popovi´c and Ney, 2007; Liu and Gildea, 2005); semantic (Gim´enez and M`arquez, 2007; Lo et al., 2012); and discourse (Comelles et al., 2010; Wong and Kit, 2012; Guzm´an et al., 2014; Joty et al., 2014). Automatic MT evaluation metrics compare the output of a system to one or more human ref- erences in order to produce a similarity score. The quality of such a metric is typically judged in terms of correlation of the scores it produces with scores given by human judges. As a result, some evaluation metrics have been trained to re- produce the scores assigned by humans as closely as possible (Albrecht and Hwa, 2008). Unfortu- nately, humans have a hard time assigning an ab- solute score to …

Content truncated to 500 words