A Gated Self-attention Memory Network for Answer Selection

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5953–5959, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5953 A Gated Self-attention Memory Network for Answer Selection Tuan Lai ∗1, Quan Hung Tran ∗2, Trung Bui 2, Daisuke Kihara 1 {lai123,dkihara}@purdue.edu, {qtran,bui}@adobe.com 1 Purdue University, West Lafayette, IN 2 Adobe Research, San Jose, CA Abstract Answer selection is an important research problem, with applications in many areas. Pre- vious deep learning based approaches for the task mainly adopt the Compare-Aggregate ar- chitecture that performs word-level compari- son followed by aggregation. In this work, we take a departure from the popular Compare- Aggregate architecture, and instead, propose a new gated self-attention memory network for the task. Combined with a simple transfer learning technique from a large-scale online corpus, our model outperforms previous meth- ods by a large margin, achieving new state-of- the-art results on two standard answer selec- tion datasets: TrecQA and WikiQA. 1 Introduction and Related Work Answer selection is an important task, with appli- cations in many areas (Lai et al., 2018). Given a question and a set of candidate answers, the task is to identify the most relevant candidate. Previous work on answer selection typically relies on fea- ture engineering, linguistic tools, or external re- sources (Wang et al., 2007; Wang and Manning, 2010; Heilman and Smith, 2010; Yih et al., 2013; Yao et al., 2013). Recently, with the renaissance of neural network models, many deep learning based methods have been proposed to address the task (Tay et al., 2017b; Shen et al., 2017; Wang et al., 2017; Bian et al., 2017; Tymoshenko and Mos- chitti, 2018; Tay et al., 2018; Tayyar Madabushi et al., 2018; Yoon et al., 2019). They outperform traditional techniques. A common trait of a num- ber of these deep learning methods is the use of the Compare-Aggregate architecture (Wang and Jiang, 2017). Typically in this architecture, con- textualized vector representations of small units such as words of the question and the candidate ∗Equal contributions. The work was conducted while the first author interned at Adobe Research. are first compared and aligned. After that, these comparison results are then aggregated to calcu- late a score indicating the relevance between the question and the candidate. On standard answer selection datasets such as TrecQA (Wang et al., 2007) or WikiQA (Yang et al., 2015), Compare- Aggregate approaches achieve very competitive performance. However, they still have some limi- tations. For example, the first few layers of most previous Compare-Aggregate models encode the question-candidate pair into sequences of contex- tualized vector representations separately (Wang et al., 2017; Shen et al., 2017; Bian et al., 2017). These sequences are independent and completely ignore the information from the other sequence. In this work, we take a departure from the popular Compare-Aggregate architecture, so in- stead, we propose a mix between two very successful architectures in machine comprehen- sion and sequence modeling, the memory …

Content truncated to 500 words