Attention-over-Attention Neural Networks for Reading Comprehension

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 593–602 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1055 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 593–602 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1055 Attention-over-Attention Neural Networks for Reading Comprehension Yiming Cui†, Zhipeng Chen†, Si Wei†, Shijin Wang†, Ting Liu‡ and Guoping Hu† †Joint Laboratory of HIT and iFLYTEK, iFLYTEK Research, Beijing, China ‡Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology, Harbin, China †{ymcui,zpchen,siwei,sjwang3,gphu}@iflytek.com ‡tliu@ir.hit.edu.cn Abstract Cloze-style reading comprehension is a representative problem in mining relation- ship between document and query. In this paper, we present a simple but novel model called attention-over-attention reader for better solving cloze-style reading compre- hension task. The proposed model aims to place another attention mechanism over the document-level attention and induces “attended attention” for final answer pre- dictions. One advantage of our model is that it is simpler than related works while giving excellent performance. In addition to the primary model, we also propose an N-best re-ranking strategy to double check the validity of the candidates and further improve the performance. Experimental results show that the proposed methods significantly outperform various state-of- the-art systems by a large margin in pub- lic datasets, such as CNN and Children’s Book Test. 1 Introduction To read and comprehend the human languages are challenging tasks for the machines, which requires that the understanding of natural languages and the ability to do reasoning over various clues. Read- ing comprehension is a general problem in the real world, which aims to read and comprehend a given article or context, and answer the questions based on it. Recently, the cloze-style reading compre- hension problem has become a popular task in the community. The cloze-style query (Taylor, 1953) is a problem that to fill in an appropriate word in the given sentences while taking the context infor- mation into account. To teach the machine to do cloze-style reading comprehensions, large-scale training data is nec- essary for learning relationships between the given document and query. To create large-scale training data for neural networks, Hermann et al. (2015) released the CNN/Daily Mail news dataset, where the document is formed by the news articles and the queries are extracted from the summary of the news. Hill et al. (2015) released the Children’s Book Test dataset afterwards, where the training samples are generated from consecutive 20 sen- tences from books, and the query is formed by 21st sentence. Following these datasets, a vast va- riety of neural network approaches have been pro- posed (Kadlec et al., 2016; Cui et al., 2016; Chen et al., 2016; Dhingra et al., 2016; Sordoni et al., 2016; Trischler et al., 2016; Seo et al., 2016; Xiong et al., 2016), and most of them stem from the attention-based neural network (Bahdanau et al., 2014), which has become a stereotype in most of the NLP tasks …

Content truncated to 500 words