Multi-level Evaluation for Machine Translation

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 361–365, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. Multi-level Evaluation for Machine Translation Boxing Chen, Hongyu Guo and Roland Kuhn National Research Council Canada first.last@nrc-cnrc.gc.ca Abstract Translations generated by current statisti- cal systems often have a large variance, in terms of their quality against human ref- erences. To cope with such variation, we propose to evaluate translations using a multi-level framework. The method varies the evaluation criteria based on the clus- ters to which a translation belongs. Our experiments on the WMT metric task data show that the multi-level framework con- sistently improves the performance of two benchmarking metrics, resulting in better correlation with human judgment. 1 Introduction The aims of automatic Machine Translation (MT) evaluation metrics, which measure the quality of translations against human references, are twofold. Firstly, they enable rapid comparisons between different statistical machine translation (SMT) systems. Secondly, they are necessary to the tuning of parameter values during system trainings. To attain these goals, many machine transla- tion metrics have been introduced in recent years. For example, metrics such as BLEU (Papineni et al., 2002), NIST (Doddington, 2002), and TER (Snover et al., 2006) rely on word n-gram surface matching. Also, metrics that make use of linguis- tic resources such as synonym dictionaries, part- of-speech tagging, or paraphrasing tables, have been proposed, including Meteor (Banerjee and Lavie, 2005) and its extensions, TER-Plus (Snover et al., 2009), and TESLA (Liu et al., 2011). In ad- dition, attempts to deploy syntactic features or se- mantic information for evaluation have also been made, giving rise to the STM and DSTM (Liu and Gildea, 2005), DEPREF (Wu et al., 2013) and MEANT family (Lo and Wu, 2011) metrics. All these evaluation metrics deploy a single evaluation criterion or use the same source of in- formation to evaluate translations. Nevertheless, translations generated by current statistical sys- tems often have widely varying scores, in terms fr−en Meteor score Density en−cs Meteor score Density fr−en BLEU score Density en−cs BLEU score Density Figure 1: Distributions of translation quality. X- axis is in the range of [0,1]. word−based Fscore Syntax−based Fscore Figure 2: Clusters of translations based on quality. Both X-axis and Y-axis are in the range of [0,1]. of their quality against human references. As a re- sult, current metrics often perform better for a por- tion of translations but worse against the others. Consider, for example, two widely used metrics, namely the sentence-level Meteor and BLUE. Fig- ure 1 depicts the distributions of the two metrics’ evaluation scores, computed on system outputs for two WMT test sets, i.e., the newstest2013.fr-en and newstest2012.en-cs. As shown in Figures 1, the variances of the created evaluation scores are large across evaluation metrics as well as test sets. Such widely varying evaluation quality, how- ever, may be clustered into multiple sub-regions, as illustrated in Figure 2. Here, we sample 300 sentences from the system output of the newstest2013.fr-en test set; we depict the F- …

Content truncated to 500 words