Capturing Global Structural Information in Long Document Question Answering with Compressive Graph Selector Network

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5036 - 5047 December 7-11, 2022 ©2022 Association for Computational Linguistics Capturing Global Structural Information in Long Document Question Answering with Compressive Graph Selector Network Yuxiang Nie123, Heyan Huang123∗, Wei Wei4, Xian-Ling Mao123 1School of Computer Science and Technology, Beijing Institute of Technology 2Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications 3Beijing Institute of Technology Southeast Academy of Information Technology 4Huazhong University of Science and Technology {nieyx,hhy63,maoxl}@bit.edu.cn,weiw@hust.edu.cn Abstract Long document question answering is a chal- lenging task due to its demands for complex reasoning over long text. Previous works usu- ally take long documents as non-structured flat texts or only consider the local structure in long documents. However, these methods usually ignore the global structure of the long docu- ment, which is essential for long-range under- standing. To tackle this problem, we propose Compressive Graph Selector Network (CGSN) to capture the global structure in a compres- sive and iterative manner. The proposed model mainly focuses on the evidence selection phase of long document question answering. Specifi- cally, it consists of three modules: local graph network, global graph network and evidence memory network. Firstly, the local graph net- work builds the graph structure of the chun- ked segment in token, sentence, paragraph and segment levels to capture the short-term depen- dency of the text. Secondly, the global graph network selectively receives the information of each level from the local graph, compresses them into the global graph nodes and applies graph attention to the global graph nodes to build the long-range reasoning over the entire text in an iterative way. Thirdly, the evidence memory network is designed to alleviate the re- dundancy problem in the evidence selection by saving the selected result in the previous steps. Extensive experiments show that the proposed model outperforms previous methods on two datasets.1 1 Introduction Long document question answering (LDQA) is a task to select relevant evidence and answer ques- tions over long text (Dasigi et al., 2021). Compared to the traditional QA tasks, whose input is often ∗Corresponding author 1We have released our codes and data in https://github. com/JerrryNie/CGSN. under 512 tokens2, the input of LDQA can be more than 20K tokens. LDQA methods can be divided into two cate- gories: end-to-end methods and select-then-read methods. The end-to-end methods usually take a question and a long text as input to select evidence and produce the answer in one step. For exam- ple, Dasigi et al. (2021) use Longformer-Encoder- Decoder (LED) model to select evidence in the encoder part and generate answers in the decoder part. The select-then-read methods firstly apply an evidence selection model to obtain evidence pieces in a long document and then use an answer genera- tion model to generate answers given the evidence pieces and the question. These methods mainly fo- cus on the evidence selection phase. For example, Karpukhin et al. (2020) and Zhu et al. (2021) select paragraphs in an open …

Content truncated to 500 words