BJTU-Toshiba’s Submission to WMT22 Quality Estimation Shared Task

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 621 - 626 December 7-8, 2022 ©2022 Association for Computational Linguistics BJTU-Toshiba’s Submission to WMT22 Quality Estimation Shared Task Hui Huang† Hui Di‡ Chunyou Li† Hanming Wu† Kazushige Oushi‡ Yufeng Chen† Jian Liu† Jin’an Xu† †Beijing Key Lab of Traffic Data Analysis and Mining, Beijing Jiaotong University ‡Research&Development Center, Toshiba (China) Co., Ltd. {18112023, 21120368, 21120416, chenyf, jianliu, jaxu}@bjtu.edu.cn dihui@toshiba.com.cn, kazushige.ouchi@toshiba.co.jp Abstract This paper presents the BJTU-Toshiba joint submission for WMT 2022 quality estimation shared task. We only participate in Task 1 (quality prediction) of the shared task, focusing on the sentence-level MQM prediction. The techniques we experimented with include the integration of monolingual language models and the pre-finetuning of pre-trained represen- tations. We tried two styles of pre-finetuning, namely Translation Language Modeling and Replaced Token Detection. We demonstrate the competitiveness of our system compared to the widely adopted XLM-RoBERTa base- line. Our system is also the top-ranking system on the Sentence-level MQM Prediction for the English-German language pair1. 1 Introduction Machine translation Quality Estimation (QE) aims to evaluate the quality of machine translation auto- matically without reference. Compared with com- monly used machine translation metrics such as BLEU (Papineni et al., 2002), QE can be applica- ble to the case where references are unavailable. It has a wide range of applications in post-editing and quality control for machine translation. This paper introduces in detail the joint submis- sion of Beijing Jiaotong University and Toshiba (China) Corporation to the quality estimation shared task in the 7th Conference on Machine Translation (WMT22), and we mainly focus on the Task 1: quality prediction. This year, the quality prediction task consists of two annotations (DA and MQM) and two levels (sentence-level and word- level), and we only participate in the Sentence- level MQM prediction, of which the goal is to pre- dict the MQM score (Freitag et al., 2021) for each source-target sentence pair. Three language pairs are involved: English-German, Chinese-English 1Our codes are openly available at the public repository https://github.com/HuihuiChyan/AwesomeQE. Figure 1: The three QE architectures we adopted. and English-Russian, with roughly 10K-20k train- ing pairs provided for each direction. Our system is mainly based on the ensemble of multiple pre-trained models, both monolingual and multilingual. The monolingual models receive only the target sequence to perform regression (only es- timating the target fluency). The multilingual mod- els receive both the source and target sequence to perform regression. We also use in-domain paral- lel data to pre-finetune the pre-trained models, to adapt their representations to the target language and domain. We try two styles of pre-finetuning, namely Translation Language Model (TLM) and Replaced Token Detection (RTD). The translation language model is to predict the random masked tokens based on the concatenation of source-target pairs. The RTD is to first randomly replace some tokens by another generator, then to detect which token is replaced. Different models are ensembled to get further improvement. 621 Direction Model Type Input Spearman Pearson En-De mBERT multilingual understanding src-tgt …

Content truncated to 500 words