MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4963–4974 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4963 MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset Marina Fomicheva,1* Shuo Sun,2* Erick Fonseca,3† Chrysoula Zerva,3 Fr´ed´eric Blain,1,4 Vishrav Chaudhary,9§ Francisco Guzm´an,5 Nina Lopatina,6‡ Andr´e F. T. Martins,3,8 Lucia Specia1,7 1University of Sheffield, 2Johns Hopkins University, 3Instituto de Telecomunicac¸˜oes, 4University of Wolverhampton, 5Meta AI, 6IQT Labs, 7Imperial College London, 8Unbabel, 9Microsoft {m.fomicheva,l.specia}@sheffield.ac.uk, ssun32@jhu.edu, nina.lopatina@getspectrum.io erick.fonseca@lx.it.pt, f.blain@wlv.ac.uk, vchaudhary@microsoft.com, fguzman@fb.com, chrysoula.zerva@tecnico.ulisboa.pt, andre.martins@unbabel.com Abstract We present MLQE-PE, a new dataset for Machine Translation (MT) Quality Estimation (QE) and Automatic Post-Editing (APE). The dataset contains annotations for eleven language pairs, including both high- and low-resource languages. Specifi- cally, it is annotated for translation quality with human labels for up to 10,000 translations per language pair in the following formats: sentence-level direct assessments and post-editing effort, and word-level binary good/bad labels. Apart from the quality-related scores, each source-translation sentence pair is accompanied by the corresponding post-edited sentence, as well as titles of the articles where the sentences were extracted from, and information on the neural MT models used to translate the text. We provide a thorough description of the data collection and annotation process as well as an analysis of the annotation distribution for each language pair. We also report the performance of baseline systems trained on the MLQE-PE dataset. The dataset is freely available and has already been used for several WMT shared tasks. Keywords: Machine Translation, quality estimation, evaluation, direct assessments, post-edits 1. Introduction Translation quality estimation (QE) is the task of eval- uating a translation system’s quality without access to reference translations (Blatz et al., 2004; Specia et al., 2018b). This task has numerous applications: deciding if a sentence or document that has been automatically translated is ready to be sent to the final user or if it needs to be post-edited by a human, flagging passages with potentially critical mistakes, using it as a metric for translation quality when a human reference is not available, or in the context of computer-aided transla- tion interfaces, highlighting text that needs human re- vision and estimating the required human effort. Due to its high relevance, QE has been the subject of evaluation campaigns in the Conference for Machine Translation (WMT) since 2014 (Bojar et al., 2014; Spe- cia et al., 2018a; Fonseca et al., 2019; Specia et al., 2020), where datasets in various language pairs have been created containing source sentences, their auto- matic translations, and human post-edited text. How- ever, the currently existing data has several shortcom- ings. First, the MT system used to produce the trans- lations is not publicly available, which makes it impos- sible to develop the so-called glass-box approaches to QE and exploit model confidence (or conversely, un- *Equal contribution. †Author’s affiliation at the time of publication is Kau- fland e-commerce. Work was done while the author was at Instituto de Telecomunicac¸˜oes. ‡Author’s affiliation at the time of publication is Spec- …

Content truncated to 500 words