Transformer Dissection: An Unified Understanding for Transformer’s Attention via the Lens of Kernel

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4344–4353, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4344 Transformer Dissection: An Unified Understanding for Transformer’s Attention via the Lens of Kernel Yao-Hung Hubert Tsai1 Shaojie Bai1 Makoto Yamada34 Louis-Philippe Morency2 Ruslan Salakhutdinov1 {1Machine Learning Department,2Language Technology Institute}, Carnegie Mellon University 3Kyoto University 4RIKEN AIP {yaohungt, shaojieb, morency, rsalakhu}@cs.cmu.edu, myamada@i.kyoto-u.ac.jp Abstract Transformer is a powerful architecture that achieves superior performance on various se- quence learning tasks, including neural ma- chine translation, language understanding, and sequence prediction. At the core of the Trans- former is the attention mechanism, which con- currently processes all inputs in the streams. In this paper, we present a new formulation of attention via the lens of the kernel. To be more precise, we realize that the attention can be seen as applying kernel smoother over the inputs with the kernel scores being the simi- larities between inputs. This new formulation gives us a better way to understand individ- ual components of the Transformer’s attention, such as the better way to integrate the posi- tional embedding. Another important advan- tage of our kernel-based formulation is that it paves the way to a larger space of compos- ing Transformer’s attention. As an example, we propose a new variant of Transformer’s at- tention which models the input as a product of symmetric kernels. This approach achieves competitive performance to the current state of the art model with less computation. In our experiments, we empirically study different kernel construction strategies on two widely used tasks: neural machine translation and se- quence prediction. 1 Introduction Transformer (Vaswani et al., 2017) is a relative new architecture which outperforms traditional deep learning models such as Recurrent Neural Networks (RNNs) (Sutskever et al., 2014) and Temporal Convolutional Networks (TCNs) (Bai et al., 2018) for sequence modeling tasks across neural machine translations (Vaswani et al., 2017), language understanding (Devlin et al., 2018), se- quence prediction (Dai et al., 2019), image gener- ation (Child et al., 2019), video activity classifica- tion (Wang et al., 2018), music generation (Huang et al., 2018a), and multimodal sentiment analy- sis (Tsai et al., 2019a). Instead of performing re- currence (e.g., RNN) or convolution (e.g., TCN) over the sequences, Transformer is a feed-forward model that concurrently processes the entire se- quence. At the core of the Transformer is its at- tention mechanism, which is proposed to integrate the dependencies between the inputs. There are up to three types of attention within the full Trans- former model as exemplified with neural machine translation application (Vaswani et al., 2017): 1) Encoder self-attention considers the source sen- tence as input, generating a sequence of encoded representations, where each encoded token has a global dependency with other tokens in the in- put sequence. 2) Decoder self-attention consid- ers the target sentence (e.g., predicted target se- quence for translation) as input, generating a se- quence of decoded representations1, where …

Content truncated to 500 words