Introducing Semantics into Speech Encoders

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1132–1145 July 9-14, 2023 ©2023 Association for Computational Linguistics Pretrained Bidirectional Distillation for Machine Translation Yimeng Zhuang, Mei Tu Samsung Research China - Beijing (SRC-B) {ym.zhuang,mei.tu}@samsung.com Abstract Knowledge transfer can boost neural machine translation (NMT), for example, by finetuning a pretrained masked language model (LM). How- ever, it may suffer from the forgetting prob- lem and the structural inconsistency between pretrained LMs and NMT models. Knowl- edge distillation (KD) may be a potential so- lution to alleviate these issues, but few studies have investigated language knowledge trans- fer from pretrained language models to NMT models through KD. In this paper, we propose Pretrained Bidirectional Distillation (PBD) for NMT, which aims to efficiently transfer bidi- rectional language knowledge from masked language pretraining to NMT models. Its ad- vantages are reflected in efficiency and effec- tiveness through a globally defined and bidi- rectional context-aware distillation objective. Bidirectional language knowledge of the entire sequence is transferred to an NMT model con- currently during translation training. Specif- ically, we propose self-distilled masked lan- guage pretraining to obtain the PBD objective. We also design PBD losses to efficiently distill the language knowledge, in the form of token probabilities, to the encoder and decoder of an NMT model using the PBD objective. Ex- tensive experiments reveal that pretrained bidi- rectional distillation can significantly improve machine translation performance and achieve competitive or even better results than previous pretrain-finetune or unified multilingual trans- lation methods in supervised, unsupervised, and zero-shot scenarios. Empirically, it is con- cluded that pretrained bidirectional distillation is an effective and efficient method for trans- ferring language knowledge from pretrained language models to NMT models. 1 Introduction Initializing parameters by a pretrained masked lan- guage model (LM) (Kenton and Toutanova, 2019) is a knowledge transfer method widely applied to natural language processing tasks. Following its success, pretrained neural machine translation (NMT) models have attracted more and more re- search interest (Conneau and Lample, 2019; Song et al., 2019; Liu et al., 2020; Li et al., 2022). However, the pretrain-finetune paradigm may suffer from potential issues. As is pointed out in He et al. (2021), the finetuned model may forget some critical language generation skills learned from the pretraining phase. The catastrophic forgetting prob- lem (Kirkpatrick et al., 2017; McCloskey and Co- hen, 1989) commonly exists in transfer learning, leading to overfitting to target domains. Hu et al. (2022); Fang et al. (2022) also observe similar for- getting problems in pretrained NMT tasks. Besides, in the pretrain-finetune paradigm, model parame- ters are initialized by a pretrained model; this re- quires structure consistency (e.g., exact dimensions, layers, attention heads, etc.) between the pretrained LM and the NMT models to some extent. However, a powerful but structurally inconsistent pretrained LM may incorporate more language knowledge. Knowledge distillation (KD) (Hinton et al., 2015) may be a potential solution to alleviate these issues, but few studies investigate language knowl- edge transfer from pretrained …

Content truncated to 500 words