Unbabel’s Participation in the WMT20 Metrics Shared Task

Proceedings of the 5th Conference on Machine Translation (WMT), pages 911–920 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 911 Unbabel’s Participation in the WMT20 Metrics Shared Task Ricardo Rei Craig Stewart Ana C Farinha Alon Lavie Unbabel AI {ricardo.rei, craig.stewart, catarina.farinha, alon.lavie}@unbabel.com Abstract We present the contribution of the Unbabel team to the WMT 2020 Shared Task on Met- rics. We intend to participate on the segment- level, document-level and system-level tracks on all language pairs, as well as the “QE as a Metric” track. Accordingly, we illus- trate results of our models in these tracks with reference to test sets from the previous year. Our submissions build upon the recently pro- posed COMET framework: we train several es- timator models to regress on different human- generated quality scores and a novel ranking model trained on relative ranks obtained from Direct Assessments. We also propose a simple technique for converting segment-level predic- tions into a document-level score. Overall, our systems achieve strong results for all language pairs on previous test sets and in many cases set a new state-of-the-art. 1 Introduction In this paper we describe our submission to the WMT20 Metrics shared task. Our work is based on the COMET1 framework, as presented in Rei et al. (2020), and extended here to evaluation of MT output at segment, document and system-level, forming the basis of our submissions to the cor- responding task tracks. Recently, automatic eval- uation of MT has followed most other sub-fields in NLP with a notable interest in leveraging the power of large, pre-trained language models. Met- rics such as BERT REGRESSOR (Shimanaka et al., 2019), BERTSCORE (Zhang et al., 2020), BLEURT (Sellam et al., 2020) and our more recent COMET (Rei et al., 2020), all build upon developments in language modelling to generate automatic metrics with high correlation with human judgement. Our 1Crosslingual Optimized Metric for Evaluation of Translation hosted at: https://github.com/ Unbabel/COMET MT evaluation models follow a similar strategy, specifically utilizing the most recent iterations of the XLM-RoBERTa model presented in Conneau et al. (2020). The uniqueness of our approach comes from our inclusion of the source text as input which was demonstrated in Takahashi et al. (2020) and Rei et al. (2020) to be beneficial to the model. In our contribution to the shared task, we demonstrate methods of further exploiting information in the source text as well as a technique to fully harness the power of pre-trained language models to further improve the prediction accuracy of our evaluation framework when more than one reference transla- tion is available. For the shared task, we utilize two primary types of models built using the COMET frame- work, namely; the Estimator models, which regress directly on human scores of MT quality such as Direct Assessment; and the COMET-RANK (base) model used to rank MT outputs and systems. In addition to the models themselves, we also make the following research contributions: 1. We introduce a method for handling multiple references at inference time and …

Content truncated to 500 words