GTCOM Neural Machine Translation Systems for WMT22

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 69 - 99 December 7-8, 2022 ©2022 Association for Computational Linguistics Findings of the WMT 2022 Shared Task on Quality Estimation Chrysoula Zerva1,2, Frédéric Blain3, Ricardo Rei2,4,5, Piyawat Lertvittayakumjorn6, José G. C. de Souza4, Steffen Eger9, Diptesh Kanojia8, Duarte Alves2, Constantin Or˘asan8, Marina Fomicheva7, André F. T. Martins1,2,4 and Lucia Specia6,7 1Instituto de Telecomunicações, 2Instituto Superior Técnico, 3University of Wolverhampton 4Unbabel, 5INESC-ID, 6Imperial College London, 7University of Sheffield 8University of Surrey, 9NLLG,Technische Fakultät, Bielefeld University f.blain@wlv.ac.uk,m.fomicheva@sheffield.ac.uk, {d.kanojia, c.orasan}@surrey.ac.uk {chrysoula.zerva,ricardo.rei,duartemalves}@tecnico.ulisboa.pt, pl1515@ic.ac.uk Abstract We report the results of the WMT 2022 shared task on Quality Estimation, in which the chal- lenge is to predict the quality of the output of neural machine translation systems at the word and sentence levels, without access to refer- ence translations. This edition introduces a few novel aspects and extensions that aim to enable more fine-grained, and explainable quality esti- mation approaches. We introduce an updated quality annotation scheme using Multidimen- sional Quality Metrics to obtain sentence- and word-level quality scores for three language pairs. We also extend the Direct Assessments and post-edit data (MLQE-PE) to new language pairs: we present a novel and large dataset on English-Marathi, as well as a zero-shot test-set on English-Yoruba. Further, we include an ex- plainability sub-task for all language pairs and present a new format of a critical error detection task for two new language pairs. Participants from 11 different teams submitted altogether 991 systems to different task variants and lan- guage pairs. 1 Introduction The 11th edition of the shared task on Quality Es- timation (QE) builds on its previous editions and findings to further benchmark methods for estimat- ing the quality of neural machine translation (MT) output at run-time, without the use of reference translations. It includes (sub)tasks that consider quality of machine translations at the word and sentence levels. Over the past years, the QE field has been mov- ing towards trainable, large, multilingual models that have been shown to achieve high performance, especially at sentence-level (Specia et al., 2021). In this edition, we further expand the provided re- sources, introducing new low-resource language pairs: a large dataset of English-Marathi, suit- able for training, development and testing and a smaller test-set on English-Yoruba for zero-shot approaches. These, as well as previously published datasets for QE, rely mainly on Direct Assessments (DA)1 and post-edited translations, which provide estimates of quality either by using the human qual- ity score(s) for each segment or by estimating the distance of a translation from a human-provided correction. As these annotations can sometimes obscure the exact location and/or significance of a translation error, we wanted to investigate the feasibility and efficiency of using a more fine- grained annotation schema to obtain quality esti- mations at word- and sentence- level, namely Mul- tidimensional Quality Metrics (MQM) (Lommel et al., 2014). MQM annotations have shown to be more trustworthy for the metrics task (Freitag et al., 2021a,b), motivating us to evaluate their suitability for the …

Content truncated to 500 words