SemFace: Pre-training Encoder and Decoder with a Semantic Interface for Neural Machine Translation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4518–4527 August 1–6, 2021. ©2021 Association for Computational Linguistics 4518 SemFace: Pre-training Encoder and Decoder with a Semantic Interface for Neural Machine Translation Shuo Ren†‡∗, Long Zhou‡, Shujie Liu‡, Furu Wei‡, Ming Zhou‡, Shuai Ma† †SKLSDE Lab, Beihang University, Beijing, China ‡Microsoft Research Asia, Beijing, China †{shuoren,mashuai}@buaa.edu.cn §{long.zhou,shujliu,fuwei,mingzhou}@microsoft.com Abstract While pre-training techniques are working very well in natural language processing, how to pre-train a decoder and effectively lever- age it for neural machine translation (NMT) still remains a tricky issue. The main rea- son is that the cross-attention module between the encoder and decoder cannot be pre-trained, and the combined encoder-decoder model can- not work well in the fine-tuning stage be- cause the inputs of the decoder cross-attention come from unknown encoder outputs. In this paper, we propose a better pre-training method for NMT by defining a semantic in- terface (SemFace) between the pre-trained en- coder and the pre-trained decoder. Specifi- cally, we propose two types of semantic in- terfaces, including CL-SemFace which re- gards cross-lingual embeddings as an inter- face, and VQ-SemFace which employs vec- tor quantized embeddings to constrain the en- coder outputs and decoder inputs in the same language-independent space. We conduct mas- sive experiments on six supervised translation pairs and three unsupervised pairs. Experimen- tal results demonstrate that our proposed Sem- Face can effectively connect the pre-trained en- coder and decoder, and achieves significant im- provement by 3.7 and 1.5 BLEU points on the two tasks respectively compared with previous pre-training-based NMT models. 1 Introduction In recent years, pre-trained language models (Pe- ters et al., 2018; Devlin et al., 2018; Radford et al., 2019; Yang et al., 2019; Raffel et al., 2020) signif- icantly boost the performances of various natural language processing (NLP) tasks, receiving exten- sive attention in NLP communities. Following the idea of unsupervised pre-training methods in the NLP area, several approaches (Lample and Con- neau, 2019; Zhu et al., 2020; Lewis et al., 2020; ∗Contribution during internship at MSRA. Liu et al., 2020) have been proposed to improve neural machine translation (NMT) models with pre- training by leveraging the large-scale monolingual corpora. The typical training process usually con- sists of two stages: pre-training an encoder and a decoder separately with a large monolingual corpus in a self-supervised manner, and then fine-tuning on specific NMT tasks (Lample and Conneau, 2019). The above method essentially pre-trains a BERT- like (Devlin et al., 2019) Transformer encoder, and uses it to initialize both the encoder and decoder. Although it shows promising results, pre-training decoder benefits little in their results. The po- tential reason is that the cross-attention between the encoder and decoder is not pre-trained, which is randomly initialized when they are connected for fine-tuning, resulting in a lack of semantic in- terfaces between the pre-trained encoder and de- coder. Another line of work attempts to pre-train a sequence-to-sequence model directly, e.g., MASS (Song …

Content truncated to 500 words