Are References Really Needed? Unbabel-IST 2021 Submission for the Metrics Shared Task

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 1030–1040 November 10–11, 2021. ©2021 Association for Computational Linguistics 1030 Are References Really Needed? Unbabel-IST 2021 Submission for the Metrics Shared Task Ricardo Rei1,2,4 Ana C. Farinha1 Chrysoula Zerva2,3 Daan van Stigt1 Craig Stewart1 Pedro G. Ramos1 Taisiya Glushkova2,3 André F. T. Martins1,2,3 Alon Lavie1 1Unbabel 2Instituto Superior Técnico 3Instituto de Telecomunicações 4INESC-ID 2,3,4Lisbon, Portugal {ricardo.rei, chrysoula.zerva, taisiya.glushkova, andre.t.martins}@tecnico.ulisboa.pt {catarina.farinha, daan.stigt, craig.stewart, pedro.ramos, alon.lavie}@unbabel.com Abstract In this paper, we present the joint contribu- tion of Unbabel and IST to the WMT 2021 Metrics Shared Task. With this year‘s focus on Multidimensional Quality Metric (MQM) as the ground-truth human assessment, our aim was to steer COMET towards higher cor- relations with MQM. We do so by first pre- training on Direct Assessments and then fine- tuning on z-normalized MQM scores. In our experiments we also show that reference- free COMET models are becoming competi- tive with reference-based models, even outper- forming the best COMET model from 2020 on this year‘s development data. Additionally, we present COMETINHO, a light-weight COMET model that is 19x faster on CPU than the orig- inal model, while also achieving state-of-the- art correlations with MQM. Finally, in the “QE as a metric” track, we also participated with a QE model trained using the OPENKIWI frame- work leveraging MQM scores and word-level annotations. 1 Introduction In this paper, we present the joint contribution of Unbabel and IST to the WMT 2021 Shared Task on Metrics. We participated in the segment-level and system-level tracks, as well as the “QE as a Metric” task. Similar to our participation last year (Rei et al., 2020b), most of the models are based on the COMET framework1 (Rei et al., 2020a). In last year‘s shared task (Mathur et al., 2020), COMET along with other trainable metrics such as PRISM (Thompson and Post, 2020) and BLEURT (Sellam et al., 2020) showed superior cor- relations with the Direct Assessments (DA) col- lected for the News Translation Shared Task. This 1Crosslingual Optimized Metric for Evaluation of Translation hosted at: https://github.com/ Unbabel/COMET year, we build on top of the models used last year to take into account that human assessments will be carried out using variants of the Multidi- mensional Quality Metric (MQM) (Lommel et al., 2014) framework and no longer based on DA (Gra- ham et al., 2013). For this reason, we extended our training dataset to include DA evaluations from WMT ranging 2015 to 2020, with the exception of en-de and zh-en for which we do not include the 2020 data given that the same is included in the MQM development data (Freitag et al., 2021). Finally, we fine-tuned these new models on the z- normalized MQM scores provided for this year‘s shared task. One of the remaining redeeming qualities of au- tomated metrics such as BLEU (Papineni et al., 2002) is that they are incredibly light-weight. De- spite the higher correlation with human judgement, trainable metrics tend to be slower to run. In an …

Content truncated to 500 words