Star-Transformer

Proceedings of NAACL-HLT 2019, pages 1315–1325 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1315 Star-Transformer Qipeng Guo† qpguo16@fudan.edu.cn Xipeng Qiu† ∗ xpqiu@fudan.edu.cn Pengfei Liu† pfliu14@fudan.edu.cn Yunfan Shao† yfshao15@fudan.edu.cn Xiangyang Xue† xyxue@fudan.edu.cn †Shanghai Key Laboratory of Intelligent Information Processing, Fudan University †School of Computer Science, Fudan University ‡New York University Zheng Zhang‡ zz@nyu.edu Abstract Although Transformer has achieved great suc- cesses on many NLP tasks, its heavy struc- ture with fully-connected attention connec- tions leads to dependencies on large train- ing data. In this paper, we present Star- Transformer, a lightweight alternative by care- ful sparsification. To reduce model complex- ity, we replace the fully-connected structure with a star-shaped topology, in which every two non-adjacent nodes are connected through a shared relay node. Thus, complexity is re- duced from quadratic to linear, while preserv- ing the capacity to capture both local compo- sition and long-range dependency. The ex- periments on four tasks (22 datasets) show that Star-Transformer achieved significant im- provements against the standard Transformer for the modestly sized datasets. 1 Introduction Recently, the fully-connected attention-based models, like Transformer (Vaswani et al., 2017), become popular in natural language processing (NLP) applications, notably machine translation (Vaswani et al., 2017) and language modeling (Radford et al., 2018). Some recent work also suggest that Transformer can be an alternative to recurrent neural networks (RNNs) and convo- lutional neural networks (CNNs) in many NLP tasks, such as GPT (Radford et al., 2018), BERT (Devlin et al., 2018), Transformer-XL (Dai et al., 2019) and Universal Transformer (Dehghani et al., 2018). More specifically, there are two limitations of the Transformer. First, the computation and mem- ∗Corresponding Author. ‡ work done at NYU Shanghai, now with AWS Shanghai AI Lab. h1 h2 h3 h4 h5 h6 h7 h8 s h1 h2 h3 h4 h5 h6 h7 h8 Figure 1: Left: Connections of one layer in Trans- former, circle nodes indicate the hidden states of in- put tokens. Right: Connections of one layer in Star- Transformer, the square node is the virtual relay node. Red edges and blue edges are ring and radical connec- tions, respectively. ory overhead of the Transformer are quadratic to the sequence length. This is especially problem- atic with long sentences. Transformer-XL (Dai et al., 2019) provides a solution which achieves the acceleration and performance improvement, but it is specifically designed for the language modeling task. Second, studies indicate that Transformer would fail on many tasks if the training data is limited, unless it is pre-trained on a large corpus. (Radford et al., 2018; Devlin et al., 2018). A key observation is that Transformer does not exploit prior knowledge well. For example, the local compositionality is already a robust induc- tive bias for modeling the text sequence. How- ever, the Transformer learns this bias from scratch, along with non-local compositionality, thereby in- creasing the learning cost. The key insight is then whether leveraging strong prior knowledge can help to “lighten up” the architecture. To address the above limitation, we …

Content truncated to 500 words