Tensorized Self-Attention: Efficiently Modeling Pairwise and Global Dependencies Together

Proceedings of NAACL-HLT 2019, pages 1256–1266 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1256 Tensorized Self-Attention: Efficiently Modeling Pairwise and Global Dependencies Together Tao Shen1, Tianyi Zhou2, Guodong Long1, Jing Jiang1, Chengqi Zhang1 1 Centre for Artificial Intelligence, School of Software, University of Technology Sydney 2 Paul G. Allen School of Computer Science & Engineering, University of Washington tao.shen@student.uts.edu.au, tianyizh@uw.edu, {guodong.long, jing.jiang, chengqi.zhang}@uts.edu.au Abstract Neural networks equipped with self-attention have parallelizable computation, light-weight structure, and the ability to capture both long- range and local dependencies. Further, their expressive power and performance can be boosted by using a vector to measure pair- wise dependency, but this requires to expand the alignment matrix to a tensor, which results in memory and computation bottlenecks. In this paper, we propose a novel attention mech- anism called “Multi-mask Tensorized Self- Attention” (MTSA), which is as fast and as memory-efficient as a CNN, but significantly outperforms previous CNN-/RNN-/attention- based models. MTSA 1) captures both pair- wise (token2token) and global (source2token) dependencies by a novel compatibility func- tion composed of dot-product and additive attentions, 2) uses a tensor to represent the feature-wise alignment scores for better ex- pressive power but only requires paralleliz- able matrix multiplications, and 3) combines multi-head with multi-dimensional attentions, and applies a distinct positional mask to each head (subspace), so the memory and compu- tation can be distributed to multiple heads, each with sequential information encoded in- dependently. The experiments show that a CNN/RNN-free model based on MTSA achieves state-of-the-art or competitive perfor- mance on nine NLP benchmarks with com- pelling memory- and time-efficiency. 1 Introduction Recurrent neural network (RNN) and convolu- tional neural network (CNN) have been broadly used as context fusion modules for natural language processing (NLP) tasks. Recently, RNN/CNN in conjunction with an attention mech- anism has been proven to be effective for con- textual feature modeling in a wide range of NLP tasks, including sentiment classification (Li et al., 2018), machine translation (Bahdanau et al., 2015), reading comprehension (Seo et al., 2017; Yu et al., 2018), etc. More recently, self-attention mechanisms have been developed for context fu- sion and syntactic dependency modeling with the advantage of fewer parameters, more paralleliz- able computation, and better empirical perfor- mance (Hu et al., 2017; Vaswani et al., 2017; Shen et al., 2018a). In addition, neural networks based solely on self-attention mechanisms have achieved state-of-the-art quality on many NLP tasks, e.g., machine translation (Vaswani et al., 2017), sen- tence embedding (Shen et al., 2018a) and semantic role labeling (Tan et al., 2017). Self-attention mechanisms can be categorized into two classes according to the type of depen- dency each aims to model. The first category is to- ken2token self-attention (Hu et al., 2017; Vaswani et al., 2017; Shen et al., 2018a) that captures syn- tactic dependency between every two tokens in a sequence. An efficient dot-product compatibil- ity function is usually deployed to measure this pairwise dependency (Vaswani et al., 2017). In contrast, additive compatibility function captures the …

Content truncated to 500 words