Inner Attention based Recurrent Neural Networks for Answer Selection

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1288–1297, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Inner Attention based Recurrent Neural Networks for Answer Selection Bingning Wang, Kang Liu, Jun Zhao National Laboratory of Pattern Recognition, Institute of Automation Chinese Academy of Sciences, Beijing, China {bingning.wang, kliu, jzhao}@nlpr.ia.ac.cn Abstract Attention based recurrent neural networks have shown advantages in representing natural language sentences (Hermann et al., 2015; Rockt¨aschel et al., 2015; Tan et al., 2015). Based on recurrent neural networks (RNN), external attention infor- mation was added to hidden representa- tions to get an attentive sentence represen- tation. Despite the improvement over non- attentive models, the attention mechanism under RNN is not well studied. In this work, we analyze the deficiency of tradi- tional attention based RNN models quanti- tatively and qualitatively. Then we present three new RNN models that add attention information before RNN hidden represen- tation, which shows advantage in repre- senting sentence and achieves new state- of-art results in answer selection task. 1 Introduction Answer selection (AS) is a crucial subtask of the open domain question answering (QA) problem. Given a question, the goal is to choose the an- swer from a set of pre-selected sentences (Heilman and Smith, 2010; Yao et al., 2013). Traditional AS models are based on lexical features such as parsing tree edit distance. Neural networks based models are proposed to represent the meaning of a sentence in a vector space and then compare the question and answer candidates in this hidden space (Wang and Nyberg, 2015; Feng et al., 2015), which have shown great success in AS. However, these models represent the question and sentence separately, which may ignore the information sub- ject to the question when representing the answer. For example, given a candidate answer: Michael Jordan abruptly retired from Chicago Bulls before the beginning of the 1993-94 NBA season to pursue a career in baseball. For a question: When did Michael Jordan retired from NBA? we should focus on the be- ginning of the 1993-94 in the sentence; how- ever, when we were asked: Which sports does Michael Jordan participates after his retire- ment from NBA? we should pay more attention to pursue a career in baseball. Recent years, attention based models are pro- posed in light of this purpose and have shown great success in many NLP tasks such as ma- chine translation (Bahdanau et al., 2014; Sutskever et al., 2014), question answering (Sukhbaatar et al., 2015) and recognizing textual entailments (Rockt¨aschel et al., 2015). When building the rep- resentation of a sentence, some attention informa- tion is added to the hidden state. For example, in attention based recurrent neural networks mod- els (Bahdanau et al., 2014) each time-step hidden representation is weighted by attention. Inspired by the attention mechanism, some attention-based RNN answer selection models have been proposed (Tan et al., 2015) in which the attention when com- puting answer representation is from question rep- resentation. However, in the RNN architecture, …

Content truncated to 500 words