Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain

Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain Markus Freitag(1), Ricardo Rei(2,3,4), Nitika Mathur(5,6), Chi-kiu Lo(7), Craig Stewart(2), George Foster(1), Alon Lavie(2), Ondˇrej Bojar(8) (1)Google Research (2)Unbabel (3)INESC-ID (4)Instituto Superior Técnico (5)The University of Melbourne (6)Oracle Digital Assistant (7)National Research Council Canada (NRC-CNRC) (8)Charles University Abstract This paper presents the results of the WMT21 Metrics Shared Task. Participants were asked to score the outputs of the translation sys- tems competing in the WMT21 News Transla- tion Task with automatic metrics on two dif- ferent domains: news and TED talks. All metrics were evaluated on how well they cor- relate at the system- and segment-level with human ratings. Contrary to previous years’ editions, this year we acquired our own hu- man ratings based on expert-based human eval- uation via Multidimensional Quality Metrics (MQM). This setup had several advantages: (i) expert-based evaluation has been shown to be more reliable, (ii) we were able to evaluate all metrics on two different domains using translations of the same MT systems, (iii) we added 5 additional translations com- ing from the same system during system de- velopment. In addition, we designed three challenge sets that evaluate the robustness of all automatic metrics. We present an exten- sive analysis on how well metrics perform on three language pairs: English→German, English→Russian and Chinese→English. We further show the impact of different reference translations on reference-based metrics and compare our expert-based MQM annotation with the DA scores acquired by WMT. 1 Introduction The metrics shared task1 has been a key component of WMT since 2008, serving as a way to validate the use of automatic MT evaluation metrics and driving the development of new metrics. We eval- uate reference-based automatic metrics that score MT output by comparing the MT with a reference translation generated by human translators, who are instructed to translate “from scratch” without post-editing from MT. In addition, we also invited 1http://www.statmt.org/wmt21/ metrics-task.html submissions of reference-free metrics (quality es- timation metrics or QE metrics) that compare MT outputs directly with the source segments. All met- rics are evaluated based on their agreement with human rating when scoring MT systems and hu- man translations at the system or sentence level. This year, we implemented several changes to the methodology that was followed in previous years editions of the task: • Expert-based human evaluation This year, we collected our own human ratings for selected lan- guage pairs (en→de, en→ru, zh→en) from pro- fessional translators via MQM (Lommel et al., 2014). As shown before (Freitag et al., 2021), this produces more reliable2 scores when com- pared to the DA-based human ratings acquired by the WMT News-Translation task. This step was necessary as Freitag et al. (2021) suggested that some automatic metrics already outperform (taking MQM as the golden standard) the DA- based human ratings that were usually used in the past for the metrics task and thus the DA-based ground-truth may be of lower quality than some of our submissions. …

Content truncated to 500 words