ClusterFormer: Neural Clustering Attention for Efficient and Effective Transformer

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2390 - 2402 May 22-27, 2022 c⃝2022 Association for Computational Linguistics ClusterFormer: Neural Clustering Attention for Efficient and Effective Transformer Ningning Wang1∗, Guobing Gan1∗, Peng Zhang1†, Shuai Zhang1, Junqiu Wei2, Qun Liu3, Xin Jiang3 1College of Intelligence and Computing, Tianjin University, Tianjin, China 2The Hong Kong Polytechnic University, China 3Huawei Noah’s Ark Lab, China {w_ning1215,ganguobing,pzhang,szhang96}@tju.edu.cn junwei@polyu.edu.hk,{qun.liu,Jiang.Xin}@huawei.com Abstract Recently, a lot of research has been carried out to improve the efficiency of Transformer. Among them, the sparse pattern-based method is an important branch of efficient Transform- ers. However, some existing sparse methods usually use fixed patterns to select words, with- out considering similarities between words. Other sparse methods use clustering patterns to select words, but the clustering process is separate from the training process of the tar- get task, which causes a decrease in effective- ness. To address these limitations, we design a neural clustering method, which can be seam- lessly integrated into the Self-Attention Mech- anism in Transformer. The clustering task and the target task are jointly trained and op- timized to benefit each other, leading to signif- icant effectiveness improvement. In addition, our method groups the words with strong de- pendencies into the same cluster and performs the attention mechanism for each cluster inde- pendently, which improves the efficiency. We verified our method on machine translation, text classification, natural language inference, and text matching tasks. Experimental results show that our method outperforms two typical sparse attention methods, Reformer and Rout- ing Transformer while having a comparable or even better time and memory efficiency. 1 Introduction Transformer (Vaswani et al., 2017) has been widely used and achieved state-of-the-art results in a va- riety of NLP tasks such as neural machine trans- lation (Bahdanau et al., 2015), text classification, etc. Its good effectiveness benefits from its core component Self-Attention Mechanism which can capture global dependencies well. However, the large calculation and memory cost limit the fur- ther application of Transformer on long sequence ∗The first two authors contributed equally †Corresponding Author tasks due to the complexity of O(N2d) of Self- Attention. As a result, many research works have been carried out to improve the efficiency of Trans- former (Tay et al., 2020b). These efficient Trans- formers can be roughly divided into two cate- gories: approximation-based (Tay et al., 2020a; Katharopoulos et al., 2020) and sparse pattern- based methods (Qiu et al., 2020; Ho et al., 2019; Beltagy et al., 2020; Liu et al., 2018). Regarding approximation-based methods, some works are based on low-rank approximation (Tay et al., 2020a; Wang et al., 2020) while others are based on kernels (Katharopoulos et al., 2020; Choromanski et al., 2020). Specifically, Lin- former (Wang et al., 2020) adopts a low-rank ap- proximation idea and projects the length dimension of keys and values to a lower dimension (N → k). This reduces the complexity to O(Nkd). However, the projection matrix in this method requires that all input sequences must …

Content truncated to 500 words