Results of the WMT20 Metrics Shared Task

Proceedings of the 5th Conference on Machine Translation (WMT), pages 688–725 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 688 Results of the WMT20 Metrics Shared Task Nitika Mathur The University of Melbourne nmathur@student.unimelb.edu.au Johnny Tian-Zheng Wei University of Southern California, jwei@umass.edu Markus Freitag Google Research freitag@google.com Qingsong Ma Tencent-CSIG, AI Evaluation Lab qingsong.mqs@gmail.com Ondˇrej Bojar Charles University, MFF ´UFAL bojar@ufal.mff.cuni.cz Abstract This paper presents the results of the WMT20 Metrics Shared Task. Participants were asked to score the outputs of the translation systems competing in the WMT20 News Translation Task with automatic metrics. Ten research groups submitted 27 metrics, four of which are reference-less “metrics”. In addition, we computed five baseline metrics, includ- ing SENTBLEU, BLEU, TER and CHRF us- ing the SacreBLEU scorer. All metrics were evaluated on how well they correlate at the system-, document- and segment-level with the WMT20 official human scores. We present an extensive analysis on influence of reference translations on metric reliability, how well automatic metrics score human trans- lations, and we also flag major discrepancies between metric and human scores when eval- uating MT systems. Finally, we investigate whether we can use automatic metrics to flag incorrect human ratings. 1 Introduction The metrics shared task1 has been a key component of WMT since 2008, serving as a way to validate the use of automatic MT evaluation metrics and drive the development of new metrics. We evaluate automatic metrics that score MT out- put by comparing them with a reference translation generated by human translators, who are instructed to translate “from scratch”, without post-editing from MT. In addition, following last year’s collab- oration with the WMT Quality Estimation (QE) task, we also invited submissions of reference-free metrics that compare MT outputs directly with the source segment. Similar to the last year’s editions, the source, ref- erence texts, and MT system outputs for the metric 1http://www.statmt.org/wmt20/ metrics-task.html task come from the News Translation Task (Bar- rault et al., 2020, which we denote as Findings 2020). This year, the language pairs were English ↔ Chinese, Czech, German, Inuktitut, Japanese, Polish, Russian and Tamil. We further included sys- tems participating in the WMT parallel corpus fil- tering task (Koehn et al., 2020): Khmer and Pashto to English.2 All metrics are evaluated based on their agree- ment with human evaluation. We evaluate met- rics at three levels: comparing MT systems on the entire testset, segments (either sentences or short paragraphs), and new this year, documents. We introduce document-level evaluation to incen- tivize the development of metrics that are take into account broader context of evaluated sentences or paragraphs, following the recent emergence of document-level MT techniques. Multiple References This year, we have two in- dependently generated references for English ↔ German, English ↔ Russian, and Chinese → En- glish. This lets us investigate the influence of ref- erences and the utility of multiple references. We instructed participants to score MT systems against the references individually as well as with all avail- able references. In addition, we …

Content truncated to 500 words