Alibaba-Translate China’s Submission for WMT 2022 Quality Estimation Shared Task

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 597 - 605 December 7-8, 2022 ©2022 Association for Computational Linguistics Alibaba-Translate China’s Submission for WMT 2022 Quality Estimation Shared Task Keqin Bao1,2∗ Yu Wan1,3∗ Dayiheng Liu1 Baosong Yang1 Wenqiang Lei4 Xiangnan He2 Derek F. Wong3 Jun Xie1 1DAMO Academy, Alibaba Group 2University of Science and Technology of China 3NLP2CT Lab, University of Macau 4National University of Singapore baokeqin@mail.ustc.edu.cn nlp2ct.ywan@gmail.com {liudayiheng.ldyh,yangbaosong.ybs,qingjing.xj}@alibaba-inc.com wenqianglei@gmail.com xiangnanhe@gmail.com derekfw@um.edu.mo Abstract In this paper, we present our submission to sentence-level MQM benchmark at Quality Es- timation Shared Task, named UNITE (Unified Translation Evaluation). Specifically, our sys- tems employ the framework of UNITE, which combined three types of input format during training with a pre-trained language model. First, we apply the pseudo-labeled data exam- ples for the continuously pre-training phase. Notably, to reduce the gap between pre-training and fine-tuning, we use data pruning and a ranking-based score normalization strategy. For the fine-tuning phase, we use both Direct Assessment (DA) and Multidimensional Qual- ity Metrics (MQM) data from past years’ WMT competitions. Finally, we collect the source- only evaluation results, and ensemble the pre- dictions generated by two UNITE models, whose backbones are XLM-R and INFOXLM, respectively. Results show that our models reach 1st overall ranking in the Multilingual and English-Russian settings, and 2nd over- all ranking in English-German and Chinese- English settings, showing relatively strong per- formances in this year’s quality estimation com- petition. 1 Introduction Quality Estimation (QE) aims at evaluating ma- chine translation without access to a gold-standard reference translation (Blatz et al., 2004; Specia et al., 2018). Different from other evaluation tasks (e.g., metric), QE arranges its process of evalu- ation via only accessing source input. As the performance of modern machine translation ap- proaches increase (Vaswani et al., 2017; Lin et al., 2022; Wei et al., 2022; Zhang et al., 2022), the QE systems should better quantify the agreement of cross-lingual semantics on source sentence and translation hypothesis. The evaluation paradigm ∗Equal contribution. Work was done when Keqin Bao and Yu Wan were interning at DAMO Academy, Alibaba Group. of QE shows its own potential for real-world ap- plications (Wang et al., 2021; Park et al., 2021; Specia et al., 2021). This paper describes Alibaba Translate China’s submission to the sentence-level MQM benchmark at WMT 2022 Quality Estima- tion Shared Task (Zerva et al., 2022). In recent years, pre-trained language models (PLMs) have shown their strong ability on extract- ing cross-lingual information (Conneau et al., 2020; Chi et al., 2021). To achieve a higher correlation with human ratings on the quality of translation outputs, plenty of trainable model-based QE ap- proaches appear, e.g., COMET-QE (Rei et al., 2020) and QEMIND (Wang et al., 2021). They both first derive the embeddings assigned with source and hypothesis sentence with given PLM, then predict the overall score based on their embed- dings with a followed feedforward network. Those model-based approaches have greatly facilitated the development of the QE community. However, those models can only handle source-only …

Content truncated to 500 words