COMET: A Neural Framework for MT Evaluation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2685–2702, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2685 COMET: A Neural Framework for MT Evaluation Ricardo Rei Craig Stewart Ana C Farinha Alon Lavie Unbabel AI {ricardo.rei, craig.stewart, catarina.farinha, alon.lavie}@unbabel.com Abstract We present COMET, a neural framework for training multilingual machine translation eval- uation models which obtains new state-of-the- art levels of correlation with human judge- ments. Our framework leverages recent break- throughs in cross-lingual pretrained language modeling resulting in highly multilingual and adaptable MT evaluation models that exploit information from both the source input and a target-language reference translation in order to more accurately predict MT quality. To showcase our framework, we train three mod- els with different types of human judgements: Direct Assessments, Human-mediated Trans- lation Edit Rate and Multidimensional Qual- ity Metrics. Our models achieve new state-of- the-art performance on the WMT 2019 Met- rics shared task and demonstrate robustness to high-performing systems. 1 Introduction Historically, metrics for evaluating the quality of machine translation (MT) have relied on assessing the similarity between an MT-generated hypothesis and a human-generated reference translation in the target language. Traditional metrics have focused on basic, lexical-level features such as counting the number of matching n-grams between the MT hypothesis and the reference translation. Metrics such as BLEU (Papineni et al., 2002) and METEOR (Lavie and Denkowski, 2009) remain popular as a means of evaluating MT systems due to their light-weight and fast computation. Modern neural approaches to MT result in much higher quality of translation that often deviates from monotonic lexical transfer between languages. For this reason, it has become increasingly evident that we can no longer rely on metrics such as BLEU to provide an accurate estimate of the quality of MT (Barrault et al., 2019). While an increased research interest in neural methods for training MT models and systems has resulted in a recent, dramatic improvement in MT quality, MT evaluation has fallen behind. The MT research community still relies largely on outdated metrics and no new, widely-adopted standard has emerged. In 2019, the WMT News Translation Shared Task received a total of 153 MT system submissions (Barrault et al., 2019). The Metrics Shared Task of the same year saw only 24 sub- missions, almost half of which were entrants to the Quality Estimation Shared Task, adapted as metrics (Ma et al., 2019). The findings of the above-mentioned task high- light two major challenges to MT evaluation which we seek to address herein (Ma et al., 2019). Namely, that current metrics struggle to accu- rately correlate with human judgement at seg- ment level and fail to adequately differentiate the highest performing MT systems. In this paper, we present COMET1, a PyTorch- based framework for training highly multilingual and adaptable MT evaluation models that can func- tion as metrics. Our framework takes advantage of recent breakthroughs in cross-lingual language modeling (Artetxe and Schwenk, 2019; Devlin et al., 2019; Conneau and Lample, 2019; Conneau et al., 2019) …

Content truncated to 500 words