Achieving Accurate Conclusions in Evaluation of Automatic Machine Translation Metrics

Proceedings of NAACL-HLT 2016, pages 1–10, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Achieving Accurate Conclusions in Evaluation of Automatic Machine Translation Metrics Yvette Graham School of Computing Dublin City University graham.yvette@gmail.com Qun Liu ADAPT Research Centre Dublin City University qliu@computing.dcu.ie Abstract Automatic Machine Translation metrics, such as BLEU, are widely used in empirical eval- uation as a substitute for human assessment. Subsequently, the performance of a given met- ric is measured by its strength of correlation with human judgment. When a newly pro- posed metric achieves a stronger correlation over that of a baseline, it is important to take into account the uncertainty inherent in cor- relation point estimates prior to concluding improvements in metric performance. Con- fidence intervals for correlations with human judgment are rarely reported in metric eval- uations, however, and when they have been reported, the most suitable methods have un- fortunately not been applied. For example, incorrect assumptions about correlation sam- pling distributions made in past evaluations risk over-estimation of significant differences in metric performance. In this paper, we pro- vide analysis of each of the issues that may lead to inaccuracies before providing detail of a method that overcomes previous challenges. Additionally, we propose a new method of translation sampling that in contrast achieves genuine high conclusivity in evaluation of the relative performance of metrics. 1 Introduction In empirical evaluation of Machine Translation (MT), automatic metrics are widely used as a sub- stitute for human assessment for the purpose of measuring differences in MT system performance. The performance of a newly proposed metric is it- self measured by the degree to which its automatic scores for a sample of MT systems correlate with human assessment of that same set of systems. A main venue for evaluation of MT metrics is the an- nual Workshop for Statistical Machine Translation (WMT) (Bojar et al., 2015) where large-scale hu- man evaluation takes place, primarily for the pur- pose of ranking systems competing in the transla- tion shared task, but additionally to use the resulting system rankings for evaluation of automatic metrics. Since 2014, WMT has used the Pearson correla- tion as the official measure for evaluation of metrics (Mach´aˇcek and Bojar, 2014; Stanojevi´c et al., 2015). Comparison of the performance of any two metrics involves the comparison of two Pearson correlation point estimates computed over a sample of MT sys- tems, therefore. Table 1 shows correlations with hu- man assessment of each of the metrics participat- ing in the Czech-to-English component of WMT- 14 metrics shared task, and, for example, if we wish to compare the performance of the top-performing metric, REDSYSSENT (Wu et al., 2014), with the popular metric BLEU (Papineni et al., 2001), this in- volves comparison of the correlation point estimate of REDSYSSENT, r = 0.993, with the weaker corre- lation point estimate of BLEU, r = 0.909, with both computed with reference to human assessment of a sample of 5 MT systems. When a new metric achieves a stronger …

Content truncated to 500 words