CR-Walker: Tree-Structured Graph Reasoning and Dialog Acts for Conversational Recommendation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 146–154 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 146 HETFORMER: Heterogeneous Transformer with Sparse Attention for Long-Text Extractive Summarization Ye Liu1, Jian-Guo Zhang1, Yao Wan2, Congying Xia1, Lifang He3, Philip S. Yu1 1 University of Illinois at Chicago, Chicago, IL, USA 2 Huazhong University of Science and Technology, Huhan, China 3 Lehigh University, Bethlehem, PA, USA {yliu279, jzhan51, cxia8, psyu@}@uic.edu, wanyao@hust.edu.cn, lih319@lehigh.edu Abstract To capture the semantic graph structure from raw text, most existing summarization ap- proaches are built on GNNs with a pre-trained model. However, these methods suffer from cumbersome procedures and inefficient com- putations for long-text documents. To miti- gate these issues, this paper proposes HET- FORMER, a Transformer-based pre-trained model with multi-granularity sparse attentions for long-text extractive summarization. Specif- ically, we model different types of seman- tic nodes in raw text as a potential hetero- geneous graph and directly learn heteroge- neous relationships (edges) among nodes by Transformer. Extensive experiments on both single- and multi-document summarization tasks show that HETFORMER achieves state- of-the-art performance in Rouge F1 while us- ing less memory and fewer parameters. 1 Introduction Recent years have seen a resounding success in the use of graph neural networks (GNNs) on doc- ument summarization tasks (Wang et al., 2020; Hanqi Jin, 2020), due to their ability to capture inter-sentence relationships in complex document. Since GNN requires node features and graph struc- ture as input, various methods, including extraction and abstraction (Li et al., 2020; Huang et al., 2020; Jia et al., 2020), have been proposed for learning desirable node representations from raw text. Par- ticularly, they have shown that Transformer-based pre-trained models such as BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) offer an effective way to initialize and fine tune the node representations as the input of GNN. Despite great success in combining Transformer- based pre-trained models with GNNs, all existing approaches have their limitations. The first limita- tion lies in the adaptation capability to long-text input. Most pre-trained methods truncate longer documents into a small fixed-length sequence (e.g., n = 512 tokens), as its attention mechanism re- quires a quadratic cost w.r.t. sequence length. This would lead to serious information loss (Li et al., 2020; Huang et al., 2020). The second limitation is that they use pre-trained models as a multi- layer feature extractor to learn better node features and build multi-layer GNNs on top of extracted features, which have cumbersome networks and tremendous parameters (Jia et al., 2020). Recently there have been several works focusing on reducing the computational overhead of fully- connected attention in Transformers. Especially, ETC (Ravula et al., 2020) and Longformer (Belt- agy et al., 2020) proposed to use local-global sparse attention in pre-trained models to limit each token to attend to a subset of the other tokens (Child et al., 2019), which achieves a linear computational cost of the sequence length. Although these methods have considered using local and global attentions …

Content truncated to 500 words