KU X Upstage’s Submission for the WMT22 Quality Estimation: Critical Error Detection Shared Task

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 606 - 614 December 7-8, 2022 ©2022 Association for Computational Linguistics KU X Upstage’s submission for the WMT22 Quality Estimation: Critical Error Detection Shared Task Sugyeong Eo1, Chanjun Park1,2, Hyeonseok Moon1, Jaehyung Seo1, Heuiseok Lim1∗ 1Korea University 2Upstage {djtnrud,bcj1210,glee889,seojae777,limhseok}@korea.ac.kr chanjun.park@upstage.ai Abstract This paper presents KU X Upstage’s submis- sion to the quality estimation (QE): critical er- ror detection (CED) shared task in WMT22. We leverage the XLM-RoBERTa large model without utilizing any additional parallel data. To the best of our knowledge, we apply prompt- based fine-tuning to the QE task for the first time. To maximize the model’s language under- standing capability, we reformulate the CED task to be similar to the masked language model objective, which is a pre-training strategy of the language model. We design intuitive templates and label words, and include auxiliary descrip- tions such as demonstration or Google Trans- late results in the input sequence. We further improve the performance through the template ensemble, and as a result of the shared task, our approach achieve the best performance for both English-German and Portuguese-English language pairs in an unconstrained setting. 1 Introduction This paper presents our submission to the critical error detection (CED) shared task among the qual- ity estimation (QE) tasks of WMT22 (Zerva et al., 2022). CED is a task of detecting cases where translation errors in source sentences or transla- tion results distort meaning in terms of race, gen- der, safety, law, finance, etc. (Specia et al., 2021; Rubino et al., 2021; Jiang et al., 2021). Critical translation errors in the shared task appear in the form of mistranslation, hallucination, and deletion in source sentences or translation results, and er- rors can be classified into five categories: additions, deletions, named entities, meaning, and numbers. Even if machine translation (MT) systems produce fluent translations, the fact that they cannot be free from fatal semantic errors emphasizes the impor- tance of preventing social repercussions from the er- rors. Forbidding socially bad influences and losses ∗ Corresponding Author from these meaning deviations is the purpose of the CED task (Specia et al., 2021). Participating systems distinguish only critical er- rors, not correct translations or simple translation errors. In contrast to last year, submissions should be provided with continuous scores rather than bi- nary labels. The official script calculates scores with automatically assigned classes based on the threshold value of the index corresponding to the number of errors. Similar to last year, we partic- ipated in unconstrained English-German (En-De) and Portuguese-English (Pt-En) utilizing released training datasets1. To perform the CED task, we exploit the XLM- RoBERTa large model (Conneau et al., 2019) as uti- lized in the baseline without additional parallel data. In addition, we adopt prompt-based fine-tuning to mitigate catastrophic forgetting during fine-tuning by maximizing the linguistic capability obtained through pre-training. In prompt-based fine-tuning, the downstream task is reformulated into a cloze- style, which is consistent with the masked language modeling objective. The word for the masked …

Content truncated to 500 words