Intermediate Self-supervised Learning for Machine Translation Quality Estimation

Proceedings of the 28th International Conference on Computational Linguistics, pages 4355–4360 Barcelona, Spain (Online), December 8-13, 2020 4355 Intermediate Self-supervised Learning for Machine Translation Quality Estimation Raphael Rubino Eiichiro Sumita National Institute of Information and Communications Technology 3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan {raphael.rubino,eiichiro.sumita}@nict.go.jp Abstract Pre-training sentence encoders is effective in many natural language processing tasks including machine translation (MT) quality estimation (QE), due partly to the scarcity of annotated QE data required for supervised learning. In this paper, we investigate the use of an intermediate self-supervised learning task for sentence encoder aiming at improving QE performances at the sentence and word levels. Our approach is motivated by a problem inherent to QE: mistakes in translation caused by wrongly inserted and deleted tokens. We modify the translation language model (TLM) training objective of the cross-lingual language model (XLM) to orientate the pre- trained model towards the target task. The proposed method does not rely on annotated data and is complementary to QE methods involving pre-trained sentence encoders and domain adapta- tion. Experiments on English-to-German and English-to-Russian translation directions show that intermediate learning improves over domain adaptated models. Additionally, our method reaches results in par with state-of-the-art QE models without requiring the combination of several ap- proaches and outperforms similar methods based on pre-trained sentence encoders. 1 Introduction Machine translation (MT) quality estimation (QE) (Blatz et al., 2003; Quirk, 2004; Specia et al., 2009) aims at evaluating the quality of translation system outputs without relying on translation references, which are required by automatic evaluation metrics such as BLEU (Papineni et al., 2002) or TER (Snover et al., 2006). Current state-of-the-art (SotA) QE approaches have switched from hand-crafted features to large data-driven neural-based models (Bojar et al., 2016). Best performing QE methods from the latest WMT QE shared task (Fonseca et al., 2019) are based on two approaches: predictor–estimator (Kim et al., 2017) and QE-specific output layers on top of pre-trained contextual embeddings (Kim et al., 2019). While both approaches make use of sentence encoder models, such as BERT (Devlin et al., 2019) or XLM (Conneau and Lample, 2019), only the second approach allows for straightforward end-to-end learning and direct fine-tuning of the pre-trained language model. However, fine-tuning pre-trained models is highly unstable when the dataset is small (Devlin et al., 2019; Zhang et al., 2020), which is the case in QE for MT as annotated datasets are scarse. To provide a smooth transition between pre-training and fine-tuning, an intermediate training step has been pro- posed (Phang et al., 2018), using large scale labeled data relevant to the target task. This approach is nonetheless limited by its reliance on annotated data for supervised learning. In this paper, we focus on providing a novel self-supervised intermediate training approach to adapt a pre-trained model to QE by modifying the popular masked LM objective. Our model is based on the translation language model of XLM with a novel training objective which simulate common mistakes observed in translations, namely deletions and insertions in translations. Our …

Content truncated to 500 words