Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1754–1765 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1754 Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation Mozhdeh Gheini, Xiang Ren, Jonathan May Information Sciences Institute University of Southern California {gheini, xiangren, jonmay}@isi.edu Abstract We study the power of cross-attention in the Transformer architecture within the context of transfer learning for machine translation, and extend the findings of studies into cross- attention when training from scratch. We conduct a series of experiments through fine- tuning a translation model on data where ei- ther the source or target language has changed. These experiments reveal that fine-tuning only the cross-attention parameters is nearly as ef- fective as fine-tuning all parameters (i.e., the entire translation model). We provide insights into why this is the case and observe that lim- iting fine-tuning in this manner yields cross- lingually aligned embeddings. The implica- tions of this finding for researchers and practi- tioners include a mitigation of catastrophic for- getting, the potential for zero-shot translation, and the ability to extend machine translation models to several new language pairs with re- duced parameter storage overhead.1 1 Introduction The Transformer (Vaswani et al., 2017) has become the de facto architecture to use across tasks with sequential data. It has been dominantly used for natural language tasks, and has more recently also pushed the state-of-the-art on vision tasks (Doso- vitskiy et al., 2021). In particular, transfer learn- ing from large pretrained Transformer-based lan- guage models has been widely adopted to train new models: adapting models such as BERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) for encoder-only tasks and models such as BART (Lewis et al., 2020) and mBART (Liu et al., 2020) for encoder-decoder tasks like machine translation (MT). This transfer learning is predominantly per- formed in the form of fine-tuning: using the values of several hundred million parameters from the pre- trained model to initialize a model and start training from there. 1Our code is available at https://github.com/ MGheini/xattn-transfer-for-mt. Fine-tuning pretrained models often involves up- dating all parameters of the model without making a distinction between them based on their impor- tance. However, copious recent studies have looked into the relative cruciality of multi-headed self- and cross- attention layers when training an MT model from scratch (Voita et al., 2019; Michel et al., 2019; You et al., 2020). Cross-attention (also known as encoder-decoder attention) layers are more impor- tant than self-attention layers in the sense that they result in more degradation in quality when pruned, and hence, are more sensitive to pruning (Voita et al., 2019; Michel et al., 2019). Also, cross- attention cannot be replaced with hard-coded coun- terparts (e.g., an input-independent Gaussian dis- tribution) without significantly hurting the perfor- mance, while self-attention can (You et al., 2020). With the ubiquity of fine-tuning as a training tool, we find a similar investigation focused on trans- fer learning missing. In this work, we inspect cross-attention and its …

Content truncated to 500 words