Discourse-Aware Semantic Self-Attention for Narrative Reading Comprehension

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2541–2552, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2541 Discourse-Aware Semantic Self-Attention for Narrative Reading Comprehension Todor Mihaylov and Anette Frank Research Training Group AIPHES Department of Computational Linguistics, Heidelberg University Heidelberg, Germany {mihaylov,frank}@cl.uni-heidelberg.de Abstract In this work, we propose to use linguistic an- notations as a basis for a Discourse-Aware Se- mantic Self-Attention encoder that we employ for reading comprehension on narrative texts. We extract relations between discourse units, events and their arguments as well as core- ferring mentions, using available annotation tools. Our empirical evaluation shows that the investigated structures improve the overall performance (up to +3.4 Rouge-L), especially intra-sentential and cross-sentential discourse relations, sentence-internal semantic role re- lations, and long-distance coreference rela- tions. We show that dedicating self-attention heads to intra-sentential relations and relations connecting neighboring sentences is benefi- cial for finding answers to questions in longer contexts. Our findings encourage the use of discourse-semantic annotations to enhance the generalization capacity of self-attention mod- els for reading comprehension. 1 Introduction Transformer-based self-attention models (Vaswani et al., 2017) have been shown to work well on many natural language tasks that require large- scale training data, such as Machine Translation (Vaswani et al., 2017; Dai et al., 2019), Language Modeling (Radford et al., 2018a; Devlin et al., 2019; Dai et al., 2019; Radford et al., 2019) or Reading Comprehension (Yu et al., 2018), and can even be trained to perform surprisingly well in sev- eral multi-modal tasks (Kaiser et al., 2017b). Recent work (Strubell et al., 2018) has shown that for downstream semantic tasks with much smaller datasets, such as Semantic Role Labeling (SRL) (Palmer et al., 2005), self-attention models greatly benefit from the use of linguistic informa- tion such as dependency parsing annotations. Mo- tivated by this work, we examine to what extent [Captain Picard went on a mission.] S1 [Cardassians took him as a prisoner.] S2 [Starfleet assigned Jellico as Picard‘s replacement.] S3 Q1: When was Picard taken prisoner? DiscRel: S1, S2 => Temp.Succession Coref: Picard, him Q2: Who replaced Picard as a captain? Context: A: when he was on a mission A: Jellico SRL: [Starfleet]A0 [assigned]V [Jellico]A1 [as Picard‘s replacement]A2. Figure 1: Motivational example: context and questions with required discourse and semantic annotations. we can use discourse and semantic information to extend self-attention-based neural models for a higher-level task such as Reading Comprehension. Reading Comprehension is a task that requires a model to answer natural language questions, given a text as context: a paragraph or even full documents. Many datasets have been proposed for the task, starting with a small multi-choice dataset (Richardson et al., 2013), large-scale au- tomatically created cloze-style datasets (Hermann et al., 2015; Hill et al., 2016) and big manually annotated datasets such as Onishi et al. (2016); Rajpurkar et al. (2016); Joshi et al. (2017); Ko- cisky et al. (2018). Previous research has shown that …

Content truncated to 500 words