Using Contextual Information for Machine Translation Evaluation

Using Contextual Information for Machine Translation Evaluation Marina Fomicheva, N´uria Bel IULA, Universitat Pompeu Fabra Barcelona, Spain marina.fomicheva@upf.edu, nuria.bel@upf.edu Abstract Automatic evaluation of Machine Translation (MT) is typically approached by measuring similarity between the candidate MT and a human reference translation. An important limitation of existing evaluation systems is that they are unable to distinguish candidate-reference differences that arise due to acceptable linguistic variation from the differences induced by MT errors. In this paper we present a new metric, UPF-Cobalt, that addresses this issue by taking into consideration the syntactic contexts of candidate and reference words. The metric applies a penalty when the words are similar but the contexts in which they occur are not equivalent. In this way, Machine Translations (MTs) that are different from the human translation but still essentially correct are distinguished from those that share high number of words with the reference but alter the meaning of the sentence due to translation errors. The results show that the method proposed is indeed beneficial for automatic MT evaluation. We report experiments based on two different evaluation tasks with various types of manual quality assessment. The metric significantly outperforms state-of-the-art evaluation systems in varying evaluation settings. Keywords: Machine Translation, Evaluation, Local Context, Alignment 1. Introduction Automatic evaluation of Machine Translation (MT) is based on the idea that the closer the MT output is to a human reference translation, the higher its quality. Thus, the task is typically approached by measuring some kind of similarity between the MT (also called candidate transla- tion) and a reference translation. Most widely used evalua- tion systems follow a simple strategy of counting the num- ber of matching words and word strings in the MT and a human reference. For example, the well known metric BLEU (Papineni et al., 2002) measures the number of word n-grams in the candidate translation that are also present in the reference. This approach, however, is not reliable since the same source sentence can be correctly translated in many different ways. The fact that the MT output does not match one of the possible translation options is not nec- essarily indicative of low MT quality. Substantial work has focused on improving reference-based evaluation with various strategies: use of additional ref- erences (Albrecht and Hwa, 2008; Madnani and Dorr, 2013; Fomicheva et al., 2015a), integration of linguistic information (Pad´o et al., 2009; Gim´enez and M`arquez, 2010; Comelles et al., 2012; Denkowski and Lavie, 2014; Guzm´an et al., 2014) and use of machine learning tech- niques (Gupta et al., 2015; Herrera et al., 2015). Despite important achievements, automatic evaluation is still a poor substitute for manual quality assessment. The correlation between the metrics’ scores and human judgments of trans- lation quality at sentence level continues to be low. The rea- son is that when comparing candidate and reference trans- lations, the metrics are not able to distinguish acceptable linguistic variation from the differences that are indicative of MT errors. In this work we propose to use local context to discriminate …

Content truncated to 500 words