COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578 - 585 December 7-8, 2022 ©2022 Association for Computational Linguistics COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task Ricardo Rei∗1,2,4, José G. C. de Souza1, Duarte M. Alves1,4, Chrysoula Zerva3,4 Ana C Farinha1, Taisiya Glushkova3,4, Alon Lavie1, Luisa Coheur2,4, André F. T. Martins1,3,4 1Unbabel, Lisbon, Portugal, 2INESC-ID, Lisbon, Portugal 3Instituto de Telecomunicações, Lisbon, Portugal 4Instituto Superior Técnico, University of Lisbon, Portugal Abstract In this paper, we present the joint contribution of Unbabel and IST to the WMT 2022 Met- rics Shared Task. Our primary submission – dubbed COMET-22 – is an ensemble between a COMET estimator model trained with Di- rect Assessments and a newly proposed mul- titask model trained to predict sentence-level scores along with OK/BAD word-level tags de- rived from Multidimensional Quality Metrics error annotations. These models are ensembled together using a hyper-parameter search that weights different features extracted from both evaluation models and combines them into a single score. For the reference-free evaluation we present COMETKIWI. Similarly to our pri- mary submission, COMETKIWI is an ensemble between two models. A traditional predictor- estimator model inspired by OPENKIWI and our new multitask model trained on Multidi- mensional Quality Metrics which can also be used without references. Both our submissions show improved correlations compared to state- of-the-art metrics from last year as well as in- creased robustness to critical errors. 1 Introduction Automatic metrics for Machine Translation (MT) are a fundamental component of MT research and development. While human evaluation is still of great importance, automatic metrics allow the rapid evaluation and comparison of MT systems on large collections of text and facilitate expansion to low resource languages and domains. Neural fine-tuned metrics in particular, have shown the ability to leverage large multilingual data during training to better compare and assess the quality of state- of-the-art MT models, outperforming traditional lexical-based metrics. Hence, our research is tar- geted to and guided by the advancements in these metrics. ∗* � ricardo.rei@unbabel.com This paper presents the joint contribution of Unbabel and Instituto Superior Técnico (IST) to the WMT 2022 Metrics Shared Task (Freitag et al., 2022). We participated in the segment-level and system-level tracks, as well as the “QE-as-a- Metric”. Similar to our participation last year (Rei et al., 2021), our models are based on the COMET framework1 (Rei et al., 2020a). Our efforts this year built on findings and obser- vations from our participation in the WMT 2021 Metrics Shared Task (Rei et al., 2021; Freitag et al., 2021b) to further improve COMET for the Metrics task and to increase its robustness to translation errors such as deviation from named entities, re- verse polarity and negation, deviation in numbers, etc. These types of fine-grained critical errors have been shown to be challenging for state-of-the-art metrics and QE systems (Amrhein and Sennrich, 2022; Kanojia et al., 2021). For that reason we aim to take advantage of finer-grained information, incorporating word-level supervision from Multi- dimensional Quality Metrics (MQM) annotations when …

Content truncated to 500 words