CSP:Code-Switching Pre-training for Neural Machine Translation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2624–2636, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2624 CSP: Code-Switching Pre-training for Neural Machine Translation Zhen Yang, Bojie Hu, Ambyera Han, Shen Huang and Qi Ju∗ Tencent Minority-Mandarin Translation {zieenyang, bojiehu, ambyera, springhuang, damonju}@tencent.com Abstract This paper proposes a new pre-training method, called Code-Switching Pre-training (CSP for short) for Neural Machine Trans- lation (NMT). Unlike traditional pre-training method which randomly masks some frag- ments of the input sentence, the proposed CSP randomly replaces some words in the source sentence with their translation words in the tar- get language. Specifically, we firstly perform lexicon induction with unsupervised word em- bedding mapping between the source and tar- get languages, and then randomly replace some words in the input sentence with their translation words according to the extracted translation lexicons. CSP adopts the encoder- decoder framework: its encoder takes the code- mixed sentence as input, and its decoder pre- dicts the replaced fragment of the input sen- tence. In this way, CSP is able to pre-train the NMT model by explicitly making the most of the cross-lingual alignment information ex- tracted from the source and target monolingual corpus. Additionally, we relieve the pretrain- finetune discrepancy caused by the artificial symbols like [mask]. To verify the effective- ness of the proposed method, we conduct ex- tensive experiments on unsupervised and su- pervised NMT. Experimental results show that CSP achieves significant improvements over baselines without pre-training or with other pre-training methods. 1 Introduction Neural machine translation (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014; Bahdanau et al., 2015) which typically fol- lows the encoder-decoder framework, directly ap- plies a single neural network to transform the source sentence into the target sentence. With * indicates corresponding author. tens of millions of trainable parameters in the NMT model, translation tasks are usually data- hungry, and many of them are low-resource or even zero-resource in terms of training data. Follow- ing the idea of unsupervised and self-supervised pre-training methods in the NLP area (Peters et al., 2018; Radford et al., 2018, 2019; Devlin et al., 2019; Yang et al., 2019), some works are proposed to improve the NMT model with pre- training, by making full use of the widely avail- able monolingual corpora (Lample and Conneau, 2019; Song et al., 2019b; Edunov et al., 2019; Huang et al., 2019; Wang et al., 2019; Rothe et al., 2019; Clinchant et al., 2019). Typically, two different branches of pre-training approaches are proposed for NMT: model-fusion and parameter- initialization. The model-fusion approaches seek to incorpo- rate the sentence representation provided by the pre- trained model, such as BERT, into the NMT model (Yang et al., 2019b; Clinchant et al., 2019; Weng et al., 2019; Zhu et al., 2020; Lewis et al., 2019; Liu et al., 2020). These approaches are able to leverage the publicly available pre-trained checkpoints in the website but they need to change the NMT model to fuse the sentence …

Content truncated to 500 words