Recurrent Chunking Mechanisms for Long-Text Machine Reading Comprehension

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6751–6761 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6751 Recurrent Chunking Mechanisms for Long-Text Machine Reading Comprehension Hongyu Gong1∗ Yelong Shen2† Dian Yu3 Jianshu Chen3 Dong Yu3 1University of Illinois at Urbana-Champaign, IL, USA 2Microsoft Dynamics 365 AI, Redmond, WA, USA 3Tencent AI Lab, Bellevue, WA, USA hgong6@illinois.edu yeshe@microsoft.com {yudian,jianshuchen,dyu}@tencent.com Abstract In this paper, we study machine reading com- prehension (MRC) on long texts, where a model takes as inputs a lengthy document and a question and then extracts a text span from the document as an answer. State-of-the-art models tend to use a pretrained transformer model (e.g., BERT) to encode the joint con- textual information of document and question. However, these transformer-based models can only take a fixed-length (e.g., 512) text as its input. To deal with even longer text inputs, previous approaches usually chunk them into equally-spaced segments and predict answers based on each segment independently without considering the information from other seg- ments. As a result, they may form segments that fail to cover the correct answer span or retain insufficient contexts around it, which significantly degrades the performance. More- over, they are less capable of answering ques- tions that need cross-segment information. We propose to let a model learn to chunk in a more flexible way via reinforcement learn- ing: a model can decide the next segment that it wants to process in either direction. We also employ recurrent mechanisms to enable infor- mation to flow across segments. Experiments on three MRC datasets – CoQA, QuAC, and TriviaQA – demonstrate the effectiveness of our proposed recurrent chunking mechanisms: we can obtain segments that are more likely to contain complete answers and at the same time provide sufficient contexts around the ground truth answers for better predictions. 1 Introduction Teaching machines to read, process, and compre- hend natural language is a coveted goal of ma- chine reading comprehension (MRC) problems ∗ The work was performed during an internship at Ten- cent AI Lab, Bellevue, WA, USA. † The work was performed when Yelong Shen was at Tencent AI Lab, Bellevue, WA, USA. (Hermann et al., 2015; Hill et al., 2016; Rajpurkar et al., 2016; Trischler et al., 2017; Zhang et al., 2018; Koˇcisk`y et al., 2018). Many existing MRC datasets have a similar task definition: given a doc- ument and a question, the goal is to extract a span from the document (in most cases) or instead gen- erate an abstractive answer to answer the question. There is a growing trend of building MRC read- ers (Hu et al., 2019; Xu et al., 2019; Yang et al., 2019a; Keskar et al., 2019) based on pre-trained language models (Baker et al., 2019; Yang et al., 2019b), such as GPT (Radford et al., 2018) and BERT (Devlin et al., 2019). These models typi- cally consist of a stack of transformer layers that only allow fixed-length (e.g., 512) inputs. However, it is often the …

Content truncated to 500 words