CrossQE: HW-TSC 2022 Submission for the Quality Estimation Shared Task

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 646 - 652 December 7-8, 2022 ©2022 Association for Computational Linguistics CrossQE: HW-TSC 2022 Submission for the Quality Estimation Shared Task Shimin Tao1*, Chang Su1∗, Miaomiao Ma1, Hao Yang1, Min Zhang1, Xiang Geng2, Shujian Huang2, Jiaxin Guo1, Minghan Wang1, Yinglu Li1 1Huawei Translation Services Center, China 2Nanjing University, China {taoshimin, suchang8, mamiaomiao, yanghao30, zhangmin186, guojiaxin1, wangminghan, liyinglu}@huawei.com gx@smail.nju.edu.cn, huangsj@nju.edu.cn Abstract Quality estimation (QE) is a crucial method to investigate automatic methods for estimating the quality of machine translation results with- out reference translations. This paper presents Huawei Translation Services Center’s (HW- TSC’s) work called CrossQE in WMT 2022 QE shared tasks 1 and 2, namely sentence- and word- level quality prediction and explain- able QE. CrossQE employes the framework of predictor-estimator for task 1, concretely with a pre-trained cross-lingual XLM-RoBERTa large as predictor and task-specific classifier or re- gressor as estimator. An extensive set of exper- imental results show that after adding bottle- neck adapter layer, mean teacher loss, masked language modeling task loss and MC dropout methods in CrossQE, the performance has im- proved to a certain extent. For task 2, CrossQE calculated the cosine similarity between each word feature in the target and each word fea- ture in the source by task 1 sentence-level QE system’s predictor, and used the inverse value of maximum similarity between each word in the target and the source as the word translation error risk value. Moreover, CrossQE has out- standing performance on QE test sets of WMT 2022. 1 Introduction Quality estimation (QE) is the task of evaluating a translation system’s quality without access to ref- erence translations (Specia et al., 2018). In WMT 2022 QE shared task 1, there are three tasks — Quality Prediction, Explainable QE and Critical Er- ror Detection. Each task involves several language pairs. Our team — Huawei Translation Services Center (HW-TSC) — participated in quality pre- diction and explainable QE tasks over all language pairs. This paper describes the HW-TSC’s systems ∗ Indicates equal contribution. 1https://wmt-qe-task.github.io/ called CrossQE submitted for these tasks. Some key steps are summarized as follow: • We used pre-trained Cross-lingual XLM- Roberta large (Lample and Conneau, 2019; Conneau et al., 2019) as predictor instead of RNN-based model in the two-stage Predictor- Estimator architecture (Kim et al., 2017). The task-specific classifier or regressor is used as quality estimator, and multitasks are trained at the same time. • The cross-lingual XLM-RoBERTa large model is pre-trained on large-scale parallel corpora where source and target tokens are concatenated by MLM task. Shuffling those tokens and predicting those tokens’ index by the pre-trained model as an additional pre- training task can improve the QE model’s ef- fect. • We build on the COMET architecture 2 by ex- ploring adapter layers (Houlsby et al., 2019) for quality estimation to eliminate the overfit- ting problem instead of fine-tuning the whole base pre-trained model for different NLP tasks (He et al., 2021). • In the training step, the Mean Teacher …

Content truncated to 500 words