Findings of the WMT 2021 Shared Task on Quality Estimation

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 684–725 November 10–11, 2021. ©2021 Association for Computational Linguistics 684 Findings of the WMT 2021 Shared Task on Quality Estimation Lucia Specia,1,2 Frédéric Blain,2,3 Marina Fomicheva,2 Chrysoula Zerva,4,5 Zhenhao Li,1 Vishrav Chaudhary,6 André F. T. Martins4,5,7 1Imperial College London, 2University of Sheffield, 3University of Wolverhampton, 4Instituto de Telecomunicações, 5Instituto Superior Técnico, 6Facebook AI, 7Unbabel {l.specia,m.fomicheva}@sheffield.ac.uk, f.blain@wlv.ac.uk chrysoula.zerva@tecnico.ulisboa.pt, zhenhao.li18@imperial.ac.uk, vishrav@fb.com, andre.t.martins@tecnico.ulisboa.pt Abstract We report the results of the WMT 2021 shared task on Quality Estimation, where the chal- lenge is to predict the quality of the output of neural machine translation systems at the word and sentence levels. This edition focused on two main novel additions: (i) prediction for unseen languages, i.e. zero-shot settings, and (ii) prediction of sentences with catastrophic errors. In addition, new data was released for a number of languages, especially post-edited data. Participating teams from 19 institutions submitted altogether 1263 systems to different task variants and language pairs. 1 Introduction The 10th edition of the shared task on Quality Esti- mation (QE) builds on its previous editions to fur- ther benchmark methods for estimating the quality of neural machine translation (MT) output at run- time, without the use of reference translations. It includes the (sub)tasks of word-level and sentence- level estimation. The document-level task was re- moved from this edition, since it has not attracted many participants in previous editions. Important elements introduced this year are: a new sentence- level task where sentences are annotated with a binary label reflecting whether or not they contain a critical error that could lead to catastrophic con- sequences; new test data for languages that are not covered by any training set for zero-shot prediction (both direct assessment and post-editing based la- bels, at sentence and word levels. scores instead of labels based on post-editing; a new multilingual sentence-level dataset mainly from Wikipedia arti- cles, where the source articles can be retrieved for document-wide context; the availability of NMT models to explore system-internal information for the task. In addition to advancing the state-of-the-art at all prediction levels, our main goals are: • To extend the MLQE-PE public benchmark datasets; • To investigate new language independent ap- proaches especially for zero-shot prediction; • To study the feasibility of unsupervised ap- proaches especially for zero-shot prediction; and • To create a new task focusing on critical error detection. We have three subtasks: Task 1 aims at predict- ing a variant of DA scores at sentence level (Section 2.1); Task 2 aims at predicting post-editing effort scores at both sentence and word levels, i.e. words that need editing, as well as missing words and incorrect source words (Section 2.2); Task 3 aims at predicting a binary label at sentence level to in- dicate whether the sentence contains one or more critical errors (Section 2.3). Tasks make use of large datasets annotated by professional translators with either 0-100 DA scor- ing, post-edition, or critical error flagging. The text domains and languages vary for each subtask. …

Content truncated to 500 words