Papago’s Submission for the WMT21 Quality Estimation Shared Task

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 935–940 November 10–11, 2021. ©2021 Association for Computational Linguistics 935 Papago’s Submission to the WMT21 Quality Estimation Shared Task Seunghyun S. Lim Hantae Kim Hyunjoong Kim Papago, Naver Corp. Papago, Naver Corp. Papago, Naver Corp. {shaun.lim, hantae.kim, soy.lovit}@navercorp.com Abstract toThis paper describes Papago’s submission the WMT 2021 Quality Estimation Task 1: Sentence-level Direct Assessment. Our multilingual Quality Estimation system explores the combination of Pretrained Language Models and Multi- task Learning architectures. We propose an iterative training pipeline based on pretraining with large amounts of in-domain synthetic data and finetuning with gold (labeled) data. We then compress our system via knowledge distillation in order to reduce parameters yet maintain strong performance. Our submitted multilingual systems perform competitively in multilingual and all 11 individual language pair settings including zero-shot. 1 Introduction Quality Estimation (QE) evaluates the quality of machine translated output without human refer- ence translation (Blatz et al., 2004). QE has a variety of applications in the Machine Translation (MT) pipeline and is particularly useful in indus- try settings by informing translation quality to end- users. High performance in sentence-level QE (Spe- cia et al., 2020) is achieved by building a model on top of Pretrained Language Model (PLM); XLM-RoBERTa-large (Conneau et al., 2020) per- forms particularly well as shown in previous WMT sentence-level QE Shared Task. However, such PLMs contain extremely large number of param- eters. This year’s task is different from the pre- vious years’ task as submitted systems are ranked based on both model size1 and model performance2. For concurrent work Gajbhiye et al. (2021) applies knowledge distillation (Hinton et al., 2015) from a PLM-based QE architecture to a much lighter ters.1Disk space without compression and number of parame- 2Pearson’s correlation coefficient, root mean square error (RMSE), mean absolute error (MAE). BiRNN-based architecture, reducing memory re- quirements. Data scarcity is another issue relevant to QE tasks where there are often limited amount of gold training data. Previous WMT systems in- corporate data augmentation techniques and show improvements in model performance when training with additional sources of data (Baek et al., 2020; Ranasinghe et al., 2020a). Our system builds a model on top of PLM and trains with Multi-task Learning (MTL) (Caruana, 1997). Similar to Hoang et al. (2018); Zhang et al. (2018) where back-translation is iteratively applied to the same monolingual corpus to succes- sively generate higher quality synthetic training data in the context of Neural Machine Translation (NMT), our proposed approach consists of an it- erative knowledge transfer procedure which aims to repeatedly produce better quality pseudo labels for large amounts of synthetic training data. Dur- ing the final stage of our training pipeline, knowl- edge distillation is applied from teacher to student model in order to reduce model size while main- taining competitive performance. We participate in WMT 2021 Quality Estimation (Specia et al., 2021) Task 1 for multilingual and all individual language pair settings. Our system is a single mul- tilingual sentence-level QE model that …

Content truncated to 500 words