UniTE: Unified Translation Evaluation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8117 - 8127 May 22-27, 2022 c⃝2022 Association for Computational Linguistics UniTE: Unified Translation Evaluation Yu Wana,b∗ Dayiheng Liub† Baosong Yangb Haibo Zhangb Boxing Chenb Derek F. Wongb† Lidia S. Chaoa aNLP2CT Lab, University of Macau nlp2ct.ywan@gmail.com, {derekfw,lidiasc}@umac.mo bAlibaba Group {liudayiheng.ldyh,yangbaosong.ybs,zhanhui.zhb, boxing.cbx}@alibaba-inc.com Abstract Translation quality evaluation plays a crucial role in machine translation. According to the input format, it is mainly separated into three tasks, i.e., reference-only, source-only and source-reference-combined. Recent meth- ods, despite their promising results, are specif- ically designed and optimized on one of them. This limits the convenience of these meth- ods, and overlooks the commonalities among tasks. In this paper, we propose UniTE, which is the first unified framework engaged with abilities to handle all three evaluation tasks. Concretely, we propose monotonic regional at- tention to control the interaction among in- put segments, and unified pretraining to bet- ter adapt multi-task learning. We testify our framework on WMT 2019 Metrics and WMT 2020 Quality Estimation benchmarks. Exten- sive analyses show that our single model can universally surpass various state-of-the-art or winner methods across tasks. Both source code and associated models are available at https://github.com/NLP2CT/UniTE. 1 Introduction Automatically evaluating the translation quality with the given reference segment(s), is of vital importance to identify the performance of Ma- chine Translation (MT) models (Freitag et al., 2020; Mathur et al., 2020a; Zhao et al., 2020; Kocmi et al., 2021). Based on the input contexts, trans- lation evaluation can be mainly categorized into three classes: 1) reference-only evaluation ( REF ) approaches like BLEU (Papineni et al., 2002) and BLEURT (Sellam et al., 2020a), which evaluate the hypothesis by referring the golden reference at tar- get side; 2) source-only evaluation ( SRC ) methods like YiSi-2 (Lo, 2019) and TransQuest (Ranasinghe ∗Work was done when Yu Wan was interning at DAMO Academy, Alibaba Group. †Dayiheng Liu and Derek F. Wong are co-corresponding authors. et al., 2020b), which are also referred as quality es- timation (QE). These methods estimate the quality of the hypothesis based on the source sentence with- out using references; 3) source-reference-combined evaluation ( SRC+REF ) works like COMET (Rei et al., 2020), where the evaluation exploits informa- tion from both source and reference. With the help of powerful pretrained language models (PLMs, Devlin et al., 2019; Conneau et al., 2020), model- based approaches (e.g., BLEURT, TransQuest, and COMET) have shown promising results in recent WMT competitions (Ma et al., 2019; Mathur et al., 2020b; Freitag et al., 2021; Fonseca et al., 2019; Specia et al., 2020, 2021). Nevertheless, each existing MT evaluation work is usually designed for one specific task, e.g., BLEURT is only used for REF task and can not support SRC and SRC+REF tasks. Moreover, those approaches preserve the same core – evalu- ating the quality of translation by referring to the given segments. We believe that it is valuable, as well as feasible, to unify the capabilities …

Content truncated to 500 words