NICT Kyoto Submission for the WMT’20 Quality Estimation Task: Intermediate Training for Domain and Task Adaptation

Proceedings of the 5th Conference on Machine Translation (WMT), pages 1042–1048 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 1042 NICT Kyoto Submission for the WMT’20 Quality Estimation Task: Intermediate Training for Domain and Task Adaptation Raphael Rubino National Institute of Information and Communications Technology 3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan raphael.rubino@nict.go.jp Abstract This paper describes the NICT Kyoto sub- mission for the WMT’20 Quality Estimation (QE) shared task. We participated in Task 2: Word and Sentence-level Post-editing Effort, which involved Wikipedia data and two trans- lation directions, namely English-to-German and English-to-Chinese. Our approach is based on multi-task fine-tuned cross-lingual language models (XLM), initially pre-trained and further domain-adapted through interme- diate training using the translation language model (TLM) approach complemented with a novel self-supervised learning task which aim is to model errors inherent to machine transla- tion outputs. Results obtained on both word and sentence-level QE show that the proposed intermediate training method is complemen- tary to language model domain adaptation and outperforms the fine-tuning only approach. 1 Introduction This paper presents the NICT Kyoto submission for the ninth edition of the quality estimation (QE) shared task organized at the fifth conference for machine translation (WMT’20). The goal of QE is to estimate the quality of machine translation (MT) output without using a translation reference. The system developed for the task and described in this paper is based on pre-trained cross-lingual language models (XLM) (Conneau and Lample, 2019), domain and task-adapted through intermedi- ate training (Phang et al., 2018) and fine-tuned in a multi-task fashion for the sentence and word-level QE objectives. It was shown during the QE shared task at WMT’19 (Fonseca et al., 2019) that pre-trained language models (LM) fine-tuned for QE reach state-of-the-art results at the levels of sentence and word following the predictor–estimator archi- tecture (Kim et al., 2017) or using a fully end- to-end approach (Kepler et al., 2019; Kim et al., 2019; Zhou et al., 2019). However, fine-tuning pre- trained LMs is highly unstable when the dataset used for fine-tuning is small (Devlin et al., 2019; Zhang et al., 2020), which is usually the case for QE, as annotated datasets are scarce and expensive to produce, and WMT QE datasets are no excep- tions (the shared task datasets are presented in Ta- ble 3). This fine-tuning instability might be due to neural network (NN) optimization difficulties or lack of generalization. (Mosbach et al., 2020) To reduce fine-tuning instability of pre-trained LMs, Phang et al. (2018) introduced intermediate training, using large scale labeled data relevant to the target task in order to provide the pre-trained model with a transition step towards the final task. This approach is nonetheless limited by its reliance on annotated data for supervised learning. In our work, we propose a novel self-supervised interme- diate training approach to adapt a pre-trained model to QE which does not rely on labelled data. We modify the popular masked LM objective to model simultaneously deletions and insertions in transla- tions, two error types …

Content truncated to 500 words