On the Stability of System Rankings at WMT

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 464–477 November 10–11, 2021. ©2021 Association for Computational Linguistics 464 On the Stability of System Rankings at WMT Rebecca Knowles National Research Council Canada Rebecca.Knowles@nrc-cnrc.gc.ca Abstract The current approach to collecting human judgments of machine translation quality for the news translation task at WMT – segment rating with document context – is the most recent in a sequence of changes to WMT hu- man annotation protocol. As these annotation protocols have changed over time, they have drifted away from some of the initial statisti- cal assumptions underpinning them, with con- sequences that call the validity of WMT news task system rankings into question. In simu- lations based on real data, we show that the rankings can be influenced by the presence of outliers (high- or low-quality systems), result- ing in different system rankings and cluster- ings. We also examine questions of annotation task composition and how ease or difficulty of translating different documents may influence system rankings. We provide discussion of ways to analyze these issues when considering future changes to annotation protocols. 1 Introduction At the WMT (now Conference on Machine Trans- lation) shared task on news translation, research groups build machine translation systems to accu- rately translate news data, as tested on test sets of recent news documents. The systems are clustered and ranked on their performance as judged by hu- man annotators. The way that human judgments of translation quality have been collected has varied over the course of WMT’s history. In this work, we examine how changes in the collection of human judgments over the last three years have resulted in rankings that are now less robust to the effects of outliers (high- or low- performing systems) and overall annotation task composition. We replicate the human judgment rankings from 2018-2020, perform simulations for reranking, and examine issues of annotation task composition and translation difficulty. We find that sampling sentences for annotators to annotate by document – intended as a step towards evaluat- ing sentences in context – reintroduces a known problem from the earlier era of relative rankings, namely that systems suffer or benefit in their rank- ings based on the quality of the other data being rated alongside them in the same annotation tasks. We begin with a discussion of the progression of direct assessment (DA) styles employed in WMT evaluations (§2) and how scoring is performed (§3), before delving into theoretical and practical under- standings of the z-scores used to rank systems (§4 and §5), including simulations and analysis of spe- cific examples. We also discuss issues around doc- ument distribution and translation difficulty (§6), and close with considerations for downstream im- pacts (§7) and future study (§8). 2 Historical Context In 2016, WMT added direct assessment (DA) scor- ing of system outputs as an investigatory ranking, with relative ranking (RR) remaining the official scoring mechanism (Bojar et al., 2016). In relative ranking, five system outputs for a given segment were ranked in comparison …

Content truncated to 500 words