Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task

Proceedings of the 5th Conference on Machine Translation (WMT), pages 921–927 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 921 Learning to Evaluate Translation Beyond English BLEURT Submissions to the WMT Metrics 2020 Shared Task Thibault Sellam Amy Pu∗ Hyung Won Chung† Sebastian Gehrmann {tsellam, puamy, hwchung, gehrmann}@google.com Qijun Tan Markus Freitag Dipanjan Das Ankur P. Parikh {qijuntan, freitag, dipanjand, aparikh}@google.com Google Research Abstract The quality of machine translation systems has dramatically improved over the last decade, and as a result, evaluation has become an in- creasingly challenging problem. This paper describes our contribution to the WMT 2020 Metrics Shared Task, the main benchmark for automatic evaluation of translation. We make several submissions based on BLEURT, a pre- viously published metric which uses transfer learning. We extend the metric beyond En- glish and evaluate it on 14 language pairs for which fine-tuning data is available, as well as 4 “zero-shot” language pairs, for which we have no labelled examples. Additionally, we focus on English to German and demonstrate how to combine BLEURT’s predictions with those of YISI and use alternative reference translations to enhance the performance. Empirical results show that the models achieve competitive re- sults on the WMT Metrics 2019 Shared Task, indicating their promise for the 2020 edition. 1 Introduction The recent progress in machine translation mod- els has led researchers to question the use of n- gram overlap metrics such as BLEU, which fo- cus solely on surface-level aspects of the gener- ated text, and thus may correlate poorly with hu- man evaluation (Papineni et al., 2002; Lin, 2004; Ma et al., 2019; Mathur et al., 2020; Belz and Reiter, 2006; Callison-Burch et al., 2006). This has led to a surge of interest for more flexi- ble metrics that use machine learning to capture semantic-level information (Celikyilmaz et al., 2020). Popular examples of such metrics include YISI-1 (Lo, 2019), ESIM (Mathur et al., 2019), BERTSCORE (Zhang et al., 2020), the Sentence ∗ Work done during a summer internship. Permanent email address: amy pu@brown.edu. †Work done as a member of the Google AI Residency Program. Mover’s Similarity (Zhao et al., 2019; Clark et al., 2019), and BLEURT (Sellam et al., 2020). These metrics utilize contextual embeddings from large models such as BERT (Devlin et al., 2019) which have been shown to capture linguistic information beyond surface-level aspects (Tenney et al., 2019). The WMT Metrics 2020 Shared Task is the ref- erence benchmark for evaluating these metrics in the context of machine translation. It tests the evaluation of systems that are to-English (X → En) and to other languages (X → Y), which re- quires a multilingual approach. An additional challenge for learned metrics is that human rat- ings are not available for all language pairs, and therefore, the models must use unlabeled data and perform zero-shot generalization. We describe several learned metrics based on BLEURT (Sellam et al., 2020), originally devel- oped for English data. We first extend BLEURT to the multilingual setup, and show that our …

Content truncated to 500 words