On the Sub-layer Functionalities of Transformer Decoder

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4799–4811 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4799 On the Sub-layer Functionalities of Transformer Decoder Yilin Yang∗ Oregon State University yilinyang721@gmail.com Longyue Wang Tencent AI Lab vinnylywang@tencent.com Shuming Shi Tencent AI Lab shumingshi@tencent.com Prasad Tadepalli Oregon State University tadepall@oregonstate.edu Stefan Lee Oregon State University leestef@oregonstate.edu Zhaopeng Tu Tencent AI Lab zptu@tencent.com Abstract There have been significant efforts to interpret the encoder of Transformer-based encoder- decoder architectures for neural machine trans- lation (NMT); meanwhile, the decoder re- mains largely unexamined despite its critical role. During translation, the decoder must predict output tokens by considering both the source-language text from the encoder and the target-language prefix produced in previous steps. In this work, we study how Transformer- based decoders leverage information from the source and target languages – developing a universal probe task to assess how informa- tion is propagated through each module of each decoder layer. We perform extensive ex- periments on three major translation datasets (WMT En-De, En-Fr, and En-Zh). Our anal- ysis provides insight on when and where de- coders leverage different sources. Based on these insights, we demonstrate that the resid- ual feed-forward module in each Transformer decoder layer can be dropped with minimal loss of performance – a significant reduction in computation and number of parameters, and consequently a significant boost to both train- ing and inference speed. 1 Introduction Transformer models have advanced the state-of- the-art on a variety of natural language pro- cessing (NLP) tasks, including machine transla- tion (Vaswani et al., 2017), natural language in- ference (Shen et al., 2018), semantic role label- ing (Strubell et al., 2018), and language represen- tation (Devlin et al., 2019). However, so far not much is known about the internal properties and functionalities it learns to achieve its superior per- formance, which poses significant challenges for human understanding of the model and potentially designing better architectures. ∗Work done when interning at Tencent AI Lab. Recent efforts on interpreting Transformer mod- els mainly focus on assessing the encoder represen- tations (Raganato et al., 2018; Yang et al., 2019; Tang et al., 2019a) or interpreting the multi-head self-attentions (Li et al., 2018; Voita et al., 2019; Michel et al., 2019). At the same time, there have been few attempts to interpret the decoder side, which we believe is also of great interest, and should be taken into account while explaining the encoder-decoder networks. The reasons are three- fold: (a) the decoder takes both source and target as input, and implicitly performs the functionalities of both alignment and language modeling, which are at the core of machine translation; (b) the encoder and decoder are tightly coupled in that the output of the encoder is fed to the decoder and the training signals for the encoder are back-propagated from the decoder; and (c) recent studies have shown that the boundary between the encoder and decoder is blurry, since some of the encoder functionalities can be substituted by the …

Content truncated to 500 words