The NiuTrans System for the WMT20 Quality Estimation Shared Task

Proceedings of the 5th Conference on Machine Translation (WMT), pages 1018–1023 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 1018 The NiuTrans System for the WMT20 Quality Estimation Shared Task Chi Hu† Hui Liu† Kai Feng† Chen Xu† Zefan Zhou† Shiqin Yan† Yingfeng Luo† Chenglong Wang† Xia Meng† Nuo Xu† Tong Xiao†‡ Jingbo Zhu†‡ †NLP Lab, School of Computer Science and Engineering Northeastern University, Shenyang, China ‡NiuTrans Reasearch, Shenyang, China huchinlp@gmail.com, liuhui0717@outlook.com, {xiaotong,zhujingbo}@mail.neu.edu.cn, Abstract This paper describes the submissions of the NiuTrans Team to the WMT 2020 Quality Estimation Shared Task (Specia et al., 2020). We participated in all tasks and all language pairs. We explored the combination of transfer learning, multi-task learning and model ensem- ble. Results on multiple tasks show that deep transformer machine translation models and multilingual pretraining methods significantly improve translation quality estimation perfor- mance. Our system achieved remarkable re- sults in multiple level tasks, e.g., our submis- sions obtained the best results on all tracks in the sentence-level Direct Assessment task1. 1 Introduction Quality estimation (QE) evaluates the quality of machine translation output without human refer- ence translations (Blatz et al., 2004). It has a wide range of applications in post-editing and quality control for machine translation. We participated in all tasks and language pairs at the WMT 2020 QE shared task2, including sentence-level Direct Assessment tasks, word and sentence-level post-editing effort tasks, and document-level QE tasks. We investigated trans- fer learning and ensemble methods using recently proposed multilingual pre-trained models (Devlin et al., 2019; Conneau et al., 2020) as well as deep transformer models (Wang et al., 2019a). Our main contributions are as follows: • We apply multi-phase pretraining (Gururan- gan et al., 2020) methods under both high- and low-resource settings to QE tasks. limit.1Our number of submissions exceeded the daily or total 2http://www.statmt.org/wmt20/ quality-estimation-task.html • We incorporate deep transformer NMT mod- els into QE models. • We propose a simple strategy to convert document-level tasks into word- and sentence- level tasks. • We explore effective ensemble methods for both word- and sentence-level predictions. Results on different level tasks show that our methods are very competitive. Our submissions achieved the best Pearson correlation on all lan- guage pairs of the sentence-level Direct Assess- ment task and the best results on English-Chinese post-editing effort tasks. We present methods for the sentence-level Di- rect Assessment task in §2. Then in §3 and §4, we describe our approaches to post-editing tasks and document-level tasks, respectively. System en- semble methods are discussed in §5. We show the detail of our submissions and the results in §6. We conclude and discuss future work in §7. 2 Sentence-level Direct Assessment Task The sentence-level Direct Assessment task is a new task where sentences are annotated with Direct Assessment (DA) scores by professional transla- tors rather than post-editing labels. DA scores for each sentence are rated from 0 to 100, and partici- pants are required to score sentences according to z-standardized DA scores. The DA task consists of seven tracks for …

Content truncated to 500 words