Findings of the WMT 2020 Shared Task on Quality Estimation

Proceedings of the 5th Conference on Machine Translation (WMT), pages 743–764 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 743 Findings of the WMT 2020 Shared Task on Quality Estimation Lucia Specia,1,2 Fr´ed´eric Blain,2,3 Marina Fomicheva,2 Erick Fonseca,4 Vishrav Chaudhary,5 Francisco Guzm´an,5 Andr´e F. T. Martins4,6 1Imperial College London, 2University of Sheffield, 3University of Wolverhampton, 4Instituto de Telecomunicac¸˜oes, 5Facebook AI, 6Unbabel {l.specia,m.fomicheva}@sheffield.ac.uk, f.blain@wlv.ac.uk erick.fonseca@lx.it.pt, {vishrav,fguzman}@fb.com, andre.martins@unbabel.com Abstract We report the results of the WMT20 shared task on Quality Estimation, where the chal- lenge is to predict the quality of the output of neural machine translation systems at the word, sentence and document levels. This edition included new data with open domain texts, direct assessment annotations, and multiple language pairs: English– German, English–Chinese, Russian–English, Romanian–English, Estonian–English, Sinhala–English and Nepali–English data for the sentence-level subtasks, English–German and English–Chinese for the word-level subtask, and English–French data for the document-level subtask. In addition, we made neural machine translation models available to participants. 19 participating teams from 27 institutions submitted altogether 1374 systems to different task variants and language pairs. 1 Introduction This shared task builds on its previous eight edi- tions to further examine automatic methods for estimating the quality of neural machine transla- tion (MT) output at run-time, without the use of reference translations. As in previous editions, it includes the (sub)tasks of word-level, sentence- level and document-level estimation. Important elements introduced this year are: a variant of the sentence-level task where sentences are anno- tated with direct assessment (DA)1 scores instead of labels based on post-editing; a new multilingual sentence-level dataset mainly from Wikipedia arti- cles, where the source articles can be retrieved for document-wide context; the availability of NMT 1We note that the procedure followed for our data diverges from that proposed by Graham et al. (2016) in three ways: (a) we employ fewer but professional translators to score each sentence, (b) scoring is done against the source segment (bilin- gual annotation) and not the reference, and (c) we provide translators with guidelines on the meaning of ranges of scores. models to explore system-internal information for the task. In addition to advancing the state of the art at all prediction levels, our main goals are: • To create a new set of public benchmarks for tasks in quality estimation. • To investigate models for predicting DA scores and their relationship with models trained for predicting post-editing effort, • To study the feasibility of multilingual (or even language independent) approaches to QE. • To study the influence of source-language document-level context for the task of QE. • To analyse the applicability of NMT model information for QE. We have three subtasks: Task 1 aims at predict- ing DA scores at sentence level (Section 2.1); Task 2 aims at predicting post-editing effort scores at both sentence and word levels, i.e. words that need editing, as well as missing words and incorrect source words (Section 2.2); Task 3 aims at predict- ing a score for an entire document as a function of the …

Content truncated to 500 words