Token-level Dynamic Self-Attention Network for Multi-Passage Reading Comprehension

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2252–2262 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2252 Token-level Dynamic Self-Attention Network for Multi-Passage Reading Comprehension Yimeng Zhuang, Huadong Wang Samsung Research China - Beijing (SRC-B) {ym.zhuang, huadong.wang}@samsung.com Abstract Multi-passage reading comprehension re- quires the ability to combine cross-passage in- formation and reason over multiple passages to infer the answer. In this paper, we introduce the Dynamic Self-attention Network (Dyn- SAN) for multi-passage reading comprehen- sion task, which processes cross-passage in- formation at token-level and meanwhile avoids substantial computational costs. The core module of the dynamic self-attention is a pro- posed gated token selection mechanism, which dynamically selects important tokens from a sequence. These chosen tokens will attend to each other via a self-attention mechanism to model long-range dependencies. Besides, con- volutional layers are combined with the dy- namic self-attention to enhance the model’s capacity of extracting local semantic. The experimental results show that the proposed DynSAN achieves new state-of-the-art perfor- mance on the SearchQA, Quasar-T and Wiki- Hop datasets. Further ablation study also val- idates the effectiveness of our model compo- nents. 1 Introduction As a critical approach for evaluating the ability of an intelligent agent to understand natural lan- guage, reading comprehension (RC) is a chal- lenging research direction, attracting many re- searchers’ interest. In real application scenarios, such as web search, the passages may be multiple and extended, and may be comprised of relevant and irrelevant contents. It involves the problem of multi-passage reading comprehension. In multi-passage setting, cross-passage infor- mation interaction is vital for modeling long-range dependencies, co-references between entities in different passages (Dhingra et al., 2018), cross- passage answer verification (Wang et al., 2018b), and multihop reasoning (Welbl et al., 2018), etc. Great efforts have been made to develop mod- els for multi-passage task, such as Wang et al. (2018b); Zhong et al. (2019); Dehghani et al. (2019a); Dhingra et al. (2018); De Cao et al. (2019); Song et al. (2018). The common prac- tice of these approaches is that all the embed- dings in a passage or a span are integrated into a single vector and the cross-passage information interactions are based on these coarse-grain se- mantic representations. However, it may cause potential issues. As is pointed out in Bahdanau et al. (2015); Cho et al. (2014), compressing all the necessary information into a single vector may lead to “sacrifice” some critical information due to the allocated capacity to remember other informa- tion. This problem is prevalent in Neural Machine Translation (NMT), the recent models, such as the Transformer (Vaswani et al., 2017), workaround this issue by decoding on token-level context en- codings of the source text. As such, we hypothe- size that fine-grain representations may keep pre- cise semantic information, and may be beneficial to cross-passage information interactions in RC tasks. In this paper, we focus on an architecture which deals with the cross-passage information at token-level. The proposed architecture is a variant …

Content truncated to 500 words