Simple and Effective Curriculum Pointer-Generator Networks for Reading Comprehension over Long Narratives

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4922–4931 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4922 Simple and Effective Curriculum Pointer-Generator Networks for Reading Comprehension over Long Narratives 1Yi Tay, 2Shuohang Wang, 3Luu Anh Tuan, 4Jie Fu, 5Minh C. Phan 6Xingdi Yuan, 7Jinfeng Rao∗, 8Siu Cheung Hui, 9Aston Zhang 1,5,8Nanyang Technological University 2Singapore Management University 3MIT CSAIL 4Mila, Polytechnic Montr´eal 6Microsoft Research Montr´eal 7Facebook AI 9Amazon AI 1ytay017@e.ntu.edu.sg Abstract This paper tackles the problem of reading comprehension over long narratives where documents easily span over thousands of to- kens. We propose a curriculum learning (CL) based Pointer-Generator framework for read- ing/sampling over large documents, enabling diverse training of the neural model based on the notion of alternating contextual diffi- culty. This can be interpreted as a form of domain randomization and/or generative pre- training during training. To this end, the usage of the Pointer-Generator softens the require- ment of having the answer within the context, enabling us to construct diverse training sam- ples for learning. Additionally, we propose a new Introspective Alignment Layer (IAL), which reasons over decomposed alignments using block-based self-attention. We evalu- ate our proposed method on the NarrativeQA reading comprehension benchmark, achieving state-of-the-art performance, improving exist- ing baselines by 51% relative improvement on BLEU-4 and 17% relative improvement on Rouge-L. Extensive ablations confirm the ef- fectiveness of our proposed IAL and CL com- ponents. 1 Introduction Teaching machines to read and comprehend is a fundamentally interesting and challeng- ing problem in AI research (Hermann et al., 2015; Trischler et al., 2016; Rajpurkar et al., 2016). While there have been considerable and broad improvements in reading and understand- ing textual snippets, the ability for machines to read/understand complete stories and novels is still in infancy (Koˇcisk`y et al., 2018). The challenge becomes insurmountable in lieu of not only the large context but also the intrinsic challenges of ∗Work done while at University of Maryland. narrative text which arguably requires a larger ex- tent of reasoning. As such, this motivates the in- ception of relevant, interesting benchmarks such as the NarrativeQA Reading Comprehension chal- lenge1 (Koˇcisk`y et al., 2018). The challenges of having a long context have been traditionally mitigated by a two-step ap- proach - retrieval first and then reading second (Chen et al., 2017; Wang et al., 2018; Lin et al., 2018). This difficulty mirrors the same challenges of open domain question answering, albeit intro- ducing additional difficulties due to the nature of narrative text (stories and retrieved excerpts need to be coherent). While some recent works have proposed going around by training retrieval and reading components end-to-end, this paper fol- lows the traditional paradigm with a slight twist. We train our models to be robust regardless of whatever is retrieved. This is in similar spirit to domain randomization (Tobin et al., 2017). In order to do so, we propose a diverse curricu- lum learning scheme (Bengio et al., 2009) based on two concepts of …

Content truncated to 500 words