Papago’s Submission to the WMT22 Quality Estimation Shared Task

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 627 - 633 December 7-8, 2022 ©2022 Association for Computational Linguistics Papago’s Submission to the WMT22 Quality Estimation Shared Task Seunghyun S. Lim∗ Papago, Naver Corp. shaun.lim@navercorp.com Jeonghyeok Park∗ Papago, Naver Corp. jeonghyeok.park@navercorp.com Abstract This paper describes anonymous submission to the WMT 2022 Quality Estimation shared task. We participate in Task 1: Quality Prediction for both sentence and word-level quality prediction tasks. Our system is a multi- lingual and multi-task model, whereby a single system can infer both sentence and word-level quality on multiple language pairs. Our sys- tem’s architecture consists of Pretrained Lan- guage Model (PLM) and task layers, and is jointly optimized for both sentence and word- level quality prediction tasks using multilin- gual dataset. We propose novel auxiliary tasks for training and explore diverse sources of ad- ditional data to demonstrate further improve- ments on performance. Through ablation study, we examine the effectiveness of proposed com- ponents and find optimal configurations to train our submission systems under each language pair and task settings. Finally, submission sys- tems are trained and inferenced using K-folds ensemble. Our systems greatly outperform task organizer’s baseline and achieve comparable performance against other participants’ submis- sions in both sentence and word-level quality prediction tasks. 1 Introduction Quality Estimation (QE) evaluates the quality of machine translated output without human reference translation (Blatz et al., 2004). Apart from QE models’ most obvious usage as being reference-less metrics for MT, it has variety of other applications in Machine Translation (MT) pipeline including but not limited to: parallel corpus filtering (Schwenk et al., 2021), curriculum learning (Ramnath et al., 2021) and decoding (Fernandes et al., 2022). High performance in both sentence and word- level quality prediction tasks is achieved by incor- porating PLM as part of QE model architecture as demonstrated in previous WMT QE findings *These authors contributed equally to this work (Specia et al., 2020, 2021). Previous years’ top performers generally incorporate various data aug- mentation techniques in order to account for lim- ited amount of annotated gold data (Lim et al., 2021; Chen et al., 2021). Multi-task training, en- sembling, or incorporating features extracted from external models are few other popular approaches that proved to work well (Lim et al., 2021; Chen et al., 2021; Zerva et al., 2021; Wang et al., 2021a). Our system is a multilingual and multi-task model, whereby a single system can infer both sen- tence and word-level quality on multiple language pairs. Our system’s architecture (§3.1) consists of PLM and task layers, and is jointly optimized for both sentence and word-level quality prediction tasks (§3.2.1) using multilingual dataset. We pro- pose novel auxiliary tasks (§3.2.2) for training and explore diverse sources of additional data (§3.3) to demonstrate further improvements on performance. Through ablation study (§5), we evaluate each com- ponents of our proposed model and use optimal configurations to train our submission systems un- der each language pair and task settings. Finally, submission systems are trained and inferenced us- …

Content truncated to 500 words