Gated-Attention Readers for Text Comprehension

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1832–1846 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1168 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1832–1846 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1168 Gated-Attention Readers for Text Comprehension Bhuwan Dhingra∗ Hanxiao Liu∗ Zhilin Yang William W. Cohen Ruslan Salakhutdinov School of Computer Science Carnegie Mellon University {bdhingra,hanxiaol,zhiliny,wcohen,rsalakhu}@cs.cmu.edu Abstract In this paper we study the problem of an- swering cloze-style questions over docu- ments. Our model, the Gated-Attention (GA) Reader1, integrates a multi-hop ar- chitecture with a novel attention mecha- nism, which is based on multiplicative in- teractions between the query embedding and the intermediate states of a recurrent neural network document reader. This enables the reader to build query-specific representations of tokens in the document for accurate answer selection. The GA Reader obtains state-of-the-art results on three benchmarks for this task–the CNN & Daily Mail news stories and the Who Did What dataset. The effectiveness of multi- plicative interaction is demonstrated by an ablation study, and by comparing to alter- native compositional operators for imple- menting the gated-attention. 1 Introduction A recent trend to measure progress towards ma- chine reading is to test a system’s ability to an- swer questions about a document it has to com- prehend. Towards this end, several large-scale datasets of cloze-style questions over a context document have been introduced recently, which allow the training of supervised machine learning systems (Hermann et al., 2015; Hill et al., 2016; Onishi et al., 2016). Such datasets can be eas- ily constructed automatically and the unambigu- ous nature of their queries provides an objective benchmark to measure a system’s performance at text comprehension. ∗BD and HL contributed equally to this work. 1Source code is available on github: https:// github.com/bdhingra/ga-reader Deep learning models have been shown to out- perform traditional shallow approaches on text comprehension tasks (Hermann et al., 2015). The success of many recent models can be attributed primarily to two factors: (1) Multi-hop architec- tures (Weston et al., 2015; Sordoni et al., 2016; Shen et al., 2016), allow a model to scan the doc- ument and the question iteratively for multiple passes. (2) Attention mechanisms, (Chen et al., 2016; Hermann et al., 2015) borrowed from the machine translation literature (Bahdanau et al., 2014), allow the model to focus on appropriate subparts of the context document. Intuitively, the multi-hop architecture allows the reader to incre- mentally refine token representations, and the at- tention mechanism re-weights different parts in the document according to their relevance to the query. The effectiveness of multi-hop reasoning and attentions have been explored orthogonally so far in the literature. In this paper, we focus on com- bining both in a complementary manner, by de- signing a novel attention mechanism which gates the evolving token representations across hops. More specifically, unlike existing models where the query attention is applied either token-wise (Hermann et …

Content truncated to 500 words