Pairwise Neural Machine Translation Evaluation

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 805–814, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Pairwise Neural Machine Translation Evaluation Francisco Guzm´an Shafiq Joty Llu´ıs M`arquez and Preslav Nakov ALT Research Group Qatar Computing Research Institute — HBKU, Qatar Foundation {fguzman,sjoty,lmarquez,pnakov}@qf.org.qa Abstract We present a novel framework for ma- chine translation evaluation using neural networks in a pairwise setting, where the goal is to select the better translation from a pair of hypotheses, given the reference translation. In this framework, lexical, syntactic and semantic information from the reference and the two hypotheses is compacted into relatively small distributed vector representations, and fed into a multi-layer neural network that models the interaction between each of the hypothe- ses and the reference, as well as between the two hypotheses. These compact repre- sentations are in turn based on word and sentence embeddings, which are learned using neural networks. The framework is flexible, allows for efficient learning and classification, and yields correlation with humans that rivals the state of the art. 1 Introduction Automatic machine translation (MT) evaluation is a necessary step when developing or comparing MT systems. Reference-based MT evaluation, i.e., comparing the system output to one or more hu- man reference translations, is the most common approach. Existing MT evaluation measures typ- ically output an absolute quality score by com- puting the similarity between the machine and the human translations. In the simplest case, the similarity is computed by counting word n-gram matches between the translation and the reference. This is the case of BLEU (Papineni et al., 2002), which has been the standard for MT evaluation for years. Nonetheless, more recent evaluation mea- sures take into account various aspects of linguis- tic similarity, and achieve better correlation with human judgments. Having absolute quality scores at the sentence level allows to rank alternative translations for a given source sentence. This is useful, for instance, for statistical machine translation (SMT) parame- ter tuning, for system comparison, and for assess- ing the progress during MT system development. The quality of automatic MT evaluation metrics is usually assessed by computing their correlation with human judgments. To that end, quality rank- ings of alternative translations have been created by human judges. It is known that assigning an absolute score to a translation is a difficult task for humans. Hence, ranking-based evaluations, where judges are asked to rank the output of 2 to 5 systems, have been used in recent years, which has yielded much higher inter-annotator agree- ment (Callison-Burch et al., 2007). These human quality judgments can be used to train automatic metrics. This supervised learning can be oriented to predict absolute scores, e.g., us- ing regression (Albrecht and Hwa, 2008), or rank- ings (Duh, 2008; Song and Cohn, 2011). A partic- ular case of the latter is used to learn in a pair- wise setting, i.e., given a reference and two al- ternative translations (or …

Content truncated to 500 words