Tencent submission for WMT20 Quality Estimation Shared Task

Proceedings of the 5th Conference on Machine Translation (WMT), pages 1062–1067 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 1062 Tencent submission for WMT20 Quality Estimation Shared Task Haijiang Wu Zixuan Wang Qingsong Ma Xinjie Wen Ruichen Wang Xiaoli Wang Yulin Zhang Zhipeng Yao Siyao Peng PCG & CSIG, Tencent Inc, China {harywu, zackiewang, qingsongma, jasonxjwen, ruichenwang, evexlwang, elwinzhang, neokevinyao, logansypeng}@tencent.com Abstract This paper presents Tencent’s submission to the WMT20 Quality Estimation (QE) Shared Task: Sentence-Level Post-editing Effort for English-Chinese in Task 2. Our system en- sembles two architectures, XLM-based and Transformer-based Predictor-Estimator mod- els. For the XLM-based Predictor-Estimator architecture, the predictor produces two types of contextualized token representations, i.e., masked XLM and non-masked XLM; the LSTM-estimator and Transformer-estimator employ two effective strategies, top-K and multi-head attention, to enhance the sentence feature representation. For Transformer-based Predictor-Estimator architecture, we improve a top-performing model by conducting three modifications: using multi-decoding in ma- chine translation module, creating a new model by replacing the transformer-based pre- dictor with XLM-based predictor, and finally integrating two models by a weighted aver- age. Our submission achieves a Pearson corre- lation of 0.664, ranking first (tied) on English- Chinese (Specia et al., 2020). 1 Introduction The development of Machine Translation (MT) re- quires efficient quality evaluation of the outputs. The widely used MT metric BLEU (Papineni et al., 2002) satisfies this demand. However, BLEU re- quires human reference translations which costs labor and time to generate. Quality Estimation (QE) is an alternative to evaluate the quality of MT outputs with no access to reference translations (Fonseca et al., 2019; Yang et al., 2019). We participate in the sentence-level task in Task 2 of the WMT20 QE Shared Task for English– Chinese (Specia et al., 2020). The sentence-level task aims to predict the Human-targeted Transla- tion Edit Rate (HTER) (Snover et al., 2006) of the MT output, which reflects the minimal amount of edits that is needed to post-edit the MT output to an acceptable one, thus denotes the overall quality of the MT output. The classical baseline model QuEst++ (Specia et al., 2015) constructed rule-based features and employed machine learning algorithm to predict HTER scores. Recent neural networks applied the newly-emerged predictor-estimator architec- ture to QE tasks. Kim et al. (2017) proposed the first predictor-estimator model to extract feature vectors by incorporating large parallel data into a bilingual RNN model, which is subsequently fed into another bidirectional RNN model to predict QE scores. Later on, Fan et al. (2019) replaced the RNN-based predictor by a bidirectional Trans- former and added 4-dimensional mis-matching fea- tures; Wang et al. (2019) used a Transformer-DLCL based predictor; and Kepler et al. (2019a) intro- duced BERT and XLM pretrained predictors into their system. Besides the improvements on model architectures, choosing the top-performing models using ensemble techniques further improves the QE performance. For example, the submission us- ing ensemble techniques achieved the best result in the sentence-level QE sub-task in both WMT19 (Fonseca et al., 2019) and CCMT19 (Yang et al., 2019). We …

Content truncated to 500 words