Document Modeling with Graph Attention Networks for Multi-grained Machine Reading Comprehension

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6708–6718 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6708 Document Modeling with Graph Attention Networks for Multi-grained Machine Reading Comprehension Bo Zheng1∗, Haoyang Wen1, Yaobo Liang2, Nan Duan2, Wanxiang Che1†, Daxin Jiang3, Ming Zhou2, Ting Liu1 1Harbin Institute of Technology, Harbin, China 2Microsoft Research Asia, Beijing, China 3STCA NLP Group, Microsoft, Beijing, China {bzheng,hywen,car,tliu}@ir.hit.edu.cn {yalia,nanduan,djiang,mingzhou}@microsoft.com Abstract Natural Questions is a new challenging ma- chine reading comprehension benchmark with two-grained answers, which are a long answer (typically a paragraph) and a short answer (one or more entities inside the long answer). De- spite the effectiveness of existing methods on this benchmark, they treat these two sub-tasks individually during training while ignoring their dependencies. To address this issue, we present a novel multi-grained machine read- ing comprehension framework that focuses on modeling documents at their hierarchical na- ture, which are different levels of granularity: documents, paragraphs, sentences, and tokens. We utilize graph attention networks to obtain different levels of representations so that they can be learned simultaneously. The long and short answers can be extracted from paragraph- level representation and token-level represen- tation, respectively. In this way, we can model the dependencies between the two-grained an- swers to provide evidence for each other. We jointly train the two sub-tasks, and our exper- iments show that our approach significantly outperforms previous systems at both long and short answer criteria. 1 Introduction Machine reading comprehension (MRC), a task that aims to answer questions based on a given document, has been substantially advanced by re- cently released datasets and models (Rajpurkar et al., 2016; Seo et al., 2017; Xiong et al., 2017; Joshi et al., 2017; Cui et al., 2017; Devlin et al., 2019; Clark and Gardner, 2018). Natural Questions (NQ, Kwiatkowski et al., 2019), a newly released benchmark, makes it more challenging by introduc- ing much longer documents than existing datasets ∗Work was done while this author was an intern at Mi- crosoft Research Asia. †Email corresponding. Figure 1: An example from NQ dataset. and questions that are from real user queries. Be- sides, unlike conventional MRC tasks (e.g. Ra- jpurkar et al.,2016), in NQ, answers are provided in a two-grained format: long answer, which is typ- ically a paragraph, and short answers, which are typically one or more entities inside the long an- swer. Figure 1 shows an example from NQ dataset. Existing approaches on NQ have obtained promising results. For example, Kwiatkowski et al. (2019) builds a pipeline model using two sepa- rate models: the Decomposable Attention model (Parikh et al., 2016) to select a long answer, and the Document Reader model (Chen et al., 2017) to extract the short answer from the selected long an- swer. Despite the effectiveness of these approaches, they treat the long and short answer extraction as two individual sub-tasks during training and fail to model this multi-grained characteristic of this benchmark, while we argue that the two sub-tasks of NQ …

Content truncated to 500 words