Cluster-Former: Clustering-based Sparse Transformer for Question Answering

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3958–3968 August 1–6, 2021. ©2021 Association for Computational Linguistics 3958 Cluster-Former: Clustering-based Sparse Transformer for Question Answering Shuohang Wang Luowei Zhou Zhe Gan Yen-Chun Chen Yuwei Fang Siqi Sun Yu Cheng Jingjing Liu Microsoft Corporation {shuowa, luowei.zhou, zhe.gan, yen-chun.chen}@microsoft.com {yuwfan, siqi.sun, yu.cheng, jingjl}@microsoft.com Abstract Transformer has become ubiquitous in the deep learning field. One of the key ingredients that destined its success is the self-attention mechanism, which allows fully-connected con- textual encoding over input tokens. However, despite its effectiveness in modeling short se- quences, self-attention suffers when handling inputs with extreme long-range dependencies, as its complexity grows quadratically w.r.t. the sequence length. Therefore, long sequences are often encoded by Transformer in chunks using a sliding window. In this paper, we propose Cluster-Former, a novel clustering- based sparse Transformer to perform atten- tion across chunked sequences. The proposed framework is pivoted on two unique types of Transformer layer: Sliding-Window Layer and Cluster-Former Layer, which encode local se- quence information and global context jointly and iteratively. This new design allows in- formation integration beyond local windows, which is especially beneficial for question an- swering (QA) tasks that rely on long-range de- pendencies. Experiments show that Cluster- Former achieves state-of-the-art performance on several major QA benchmarks. 1 Introduction Long-range contextual understanding has proven critical in many natural language processing (NLP) tasks. For example, the relevant context for cor- rectly answering an open-domain question can arch over thousands of words (Chen et al., 2017). En- coding long sequences via deep neural networks, however, has remained an expensive and challeng- ing task due to high demand on training time and GPU memory. Traditional sequence model- ing methods (Hochreiter and Schmidhuber, 1997) encode long sequences in a chronological order, which suffers high latency. In the place of se- quential encoding, recent models such as Trans- former (Vaswani et al., 2017) use simultaneous self- attention over the entire input instead, which has been successfully adopted in many NLP tasks such as textual entailment (Devlin et al., 2019), depen- dency parsing (Zhou and Zhao, 2019), and summa- rization (Lewis et al., 2019). A caveat with Trans- former though is that building full connections over long sequences translates to quadratic growth on memory demand and computational complexity w.r.t. sequence length. One way to efficiently encode long sequences is to first chunk a sequence into much shorter ones with a sliding window, then build connections be- tween the shorter sequences (Figure 1(a)). For ex- ample, Child et al. (2019), Beltagy et al. (2020) and Zaheer et al. (2020) apply sparse attention to chun- ked sequences in hand-designed patterns in order to gather information from the chunks (Figure 1(b)). Choi et al. (2017) and Wang et al. (2019) first use a simpler model to filter chunked sequences, then process selected sequences with fully-connected self-attention. Rae et al. (2019) makes use of the shared memory of chunked sequences to build con- nections between them. However, these methods cannot encode long-range dependencies with …

Content truncated to 500 words