CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634 - 645 December 7-8, 2022 ©2022 Association for Computational Linguistics COMETKIWI: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task Ricardo Rei∗1,2,4, Marcos Treviso∗3,4, Nuno M. Guerreiro∗3,4, Chrysoula Zerva∗3,4, Ana C. Farinha1, Christine Maroti1, José G. C. de Souza1, Taisiya Glushkova3,4, Duarte M. Alves1,4, Alon Lavie1, Luisa Coheur2,4, André F. T. Martins1,3,4 1Unbabel, Lisbon, Portugal, 2INESC-ID, Lisbon, Portugal 3Instituto de Telecomunicações, Lisbon, Portugal 4Instituto Superior Técnico, University of Lisbon, Portugal Abstract We present the joint contribution of IST and Un- babel to the WMT 2022 Shared Task on Quality Estimation (QE). Our team participated on all three subtasks: (i) Sentence and Word-level Quality Prediction; (ii) Explainable QE; and (iii) Critical Error Detection. For all tasks we build on top of the COMET framework, connect- ing it with the predictor-estimator architecture of OPENKIWI, and equipping it with a word- level sequence tagger and an explanation extrac- tor. Our results suggest that incorporating refer- ences during pretraining improves performance across several language pairs on downstream tasks, and that jointly training with sentence and word-level objectives yields a further boost. Furthermore, combining attention and gradient information proved to be the top strategy for extracting good explanations of sentence-level QE models. Overall, our submissions achieved the best results for all three tasks for almost all language pairs by a considerable margin.1 1 Introduction Quality Estimation (QE) is the task of automati- cally assigning a quality score to a machine trans- lation output without depending on reference trans- lations (Specia et al., 2018). In this paper, we de- scribe the joint contribution of Instituto Superior Técnico (IST) and Unbabel to the WMT22 Quality Estimation shared task (Zerva et al., 2022), where systems were submitted to three tasks: (i) Sentence and Word-level Quality Prediction; (ii) Explainable QE; and (iii) Critical Error Detection. This year, we leverage the similarity between the tasks of MT evaluation and QE and bring together the strengths of two frameworks, COMET (Rei et al., 2020), which has been originally devel- oped for reference-based MT evaluation, and OPENKIWI (Kepler et al., 2019), which has been developed for word-level and sentence-level QE. ∗Equal contribution. � ricardo.rei@unbabel.com 1https://github.com/Unbabel/COMET Namely, we implement some of the features of the latter, as well as other new features, into the COMET framework. The result is COMETKIWI, which links the predictor-estimator architecture with COMET training-style, and incorporates word- level sequence tagging. Given that some language pairs (LPs) in the test set were not present in the training data, we aimed at developing QE systems that achieve good multi- lingual generalization and that are flexible enough to account for unseen languages through few-shot training. To do so, we start by pretraining our QE models on Direct Assessments (DAs) annotations from the previous year’s Metrics shared task as it was shown to be beneficial in our previous submis- sion (Zerva et al., 2021). Then we fine-tune our models with the data made available by the shared task.2 We experimented with different …

Content truncated to 500 words