Testing for Significance of Increased Correlation with Human Judgment

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 172–176, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Testing for Significance of Increased Correlation with Human Judgment Yvette Graham Timothy Baldwin Department of Computing and Information Systems The University of Melbourne graham.yvette@gmail.com, tb@ldwin.net Abstract Automatic metrics are widely used in ma- chine translation as a substitute for hu- man assessment. With the introduction of any new metric comes the question of just how well that metric mimics human assessment of translation quality. This is often measured by correlation with hu- man judgment. Significance tests are gen- erally not used to establish whether im- provements over existing methods such as BLEU are statistically significant or have occurred simply by chance, however. In this paper, we introduce a significance test for comparing correlations of two metrics, along with an open-source implementation of the test. When applied to a range of metrics across seven language pairs, tests show that for a high proportion of metrics, there is insufficient evidence to conclude significant improvement over BLEU. 1 Introduction Within machine translation (MT), efforts are on- going to improve evaluation metrics and find bet- ter ways to automatically assess translation qual- ity. The process of validating a new metric in- volves demonstration that it correlates better with human judgment than a standard metric such as BLEU (Papineni et al., 2001). However, although it is standard practice in MT evaluation to mea- sure increases in automatic metric scores with sig- nificance tests (Germann, 2003; Och, 2003; Ku- mar and Byrne, 2004; Koehn, 2004; Riezler and Maxwell, 2005; Graham et al., 2014), this has not been the case in papers proposing new met- rics. Thus it is possible that some reported im- provements in correlation with human judgment are attributable to chance rather than a systematic improvement. In this paper, we motivate and introduce a novel significance test to assess the statistical signifi- cance of differences in correlation with human judgment for pairs of automatic metrics. We ap- ply tests to the WMT-12 shared metrics task to compare each of the participating methods, and find that for a high proportion of metrics, there is not enough evidence to conclude that they signifi- cantly outperform BLEU. 2 Correlation with Human Judgment A common means of assessing automatic MT evaluation metrics is Spearman’s rank correlation with human judgments (Melamed et al., 2003), which measures the relative degree of monotonic- ity between the metric and human scores in the range [−1, 1]. The standard justification for cal- culating correlations over ranks rather than raw scores is to: (a) reduce anomalies due to absolute score differences; and (b) focus evaluation on what is generally the primary area of interest, namely the ranking of systems/translations. An alternative means of evaluation is Pearson’s correlation, which measures the linear correlation between a metric and human scores (Leusch et al., 2003). Debate on the relative merits of Spear- man’s and Pearson’s correlation for the evaluation of automatic metrics …

Content truncated to 500 words