RoBLEURT Submission for WMT2021 Metrics Task

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 1053–1058 November 10–11, 2021. ©2021 Association for Computational Linguistics 1053 RoBLEURT Submission for the WMT2021 Metrics Task Yu Wan1∗ Dayiheng Liu2† Baosong Yang2 Tianchi Bi2 Haibo Zhang2 Boxing Chen2 Weihua Luo2 Derek F. Wong1† Lidia S. Chao1 1NLP2CT Lab, University of Macau nlp2ct.ywan@gmail.com, {derekfw,lidiasc}@um.edu.mo 2DAMO Academy, Alibaba Group {liudayiheng.ldyh,yangbaosong.ybs,tianchi.btc, zhanhui.zhb,boxing.cbx,weihua.luowh}@alibaba-inc.com Abstract In this paper, we present our submission to Shared Metrics Task: RoBLEURT (Robustly Optimizing the training of BLEURT). After investigating the recent advances of trainable metrics, we conclude several aspects of vital importance to obtain a well-performed met- ric model by: 1) jointly leveraging the advan- tages of source-included model and reference- only model, 2) continuously pre-training the model with massive synthetic data pairs, and 3) fine-tuning the model with data denoising strat- egy. Experimental results show that our model reaching state-of-the-art correlations with the WMT2020 human annotations upon 8 out of 10 to-English language pairs. 1 Introduction Automatically evaluating the adequacy of machine translation (MT) candidates is crucial for judging the quality of MT systems. N-gram-based metrics, such as BLEU (Papineni et al., 2002), TER (Snover et al., 2006) and chrF++ (Popovic, 2015, 2017), have dominated in the topic of MT metric. De- spite the success, recent studies (Smith et al., 2016; Mathur et al., 2020a) also pointed out that, N-gram- based metrics often fail to robustly match para- phrases and capture distant dependencies. As MT systems become stronger in recent decades, these metrics show lower correlations with human judge- ments, leading the derived results unreliable. One arising direction for metric task is using trainable model to evaluate the semantic consis- tency between candidates and golden references via predicting scores. BERTScore (Zhang et al., 2020), BLEURT (Sellam et al., 2020) and COMET (Rei et al., 2020) have shown higher correlations with human judgements than N-gram-based automatic metrics. Benefiting from the powerful pre-trained ∗Work was done when Yu Wan was interning at DAMO Academy, Alibaba Group. †Corresponding authors. language models (LMs), e.g., BERT (Devlin et al., 2019), those fine-tuned metric models first derive the representation of each input, then introduce an extra linear regression module to give predicted score which describes to what degree the MT sys- tem output adequately expresses the semantic of source/reference contents. Furthermore, related work (Takahashi et al., 2020; Rei et al., 2020) re- ports that, metrics which additionally introduces source sentences into inputs can further boost the performance of metric model. To push such “model as a metric” approach fur- ther, we present RoBLEURT – Robustly optimiz- ing the training of BLEURT (Sellam et al., 2020), to achieve a better consistency between model pre- dictions and human assessments. Specifically, for low-resource scenarios, using only hypotheses and references can give more accurate results, allevi- ating the sparsity of source-side language; for the high-resource language pairs, we format the model input as the combination of source, hypothesis and reference sentences, making model attending to both source input and target reference when evaluat- ing the consistency of …

Content truncated to 500 words