Original Semantics-Oriented Attention and Deep Fusion Network for Sentence Matching

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2652–2661, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2652 Original Semantics-Oriented Attention and Deep Fusion Network for Sentence Matching Mingtong Liu, Yujie Zhang, Jinan Xu, Yufeng Chen School of Computer and Information Technology, Beijing Jiaotong University, Beijing, China {16112075,yjzhang,jaxu,chenyf}@bjtu.edu.cn Abstract Sentence matching is a key issue in natural lan- guage inference and paraphrase identification. Despite the recent progress on multi-layered neural network with cross sentence attention, one sentence learns attention to the inter- mediate representations of another sentence, which are propagated from preceding layers and therefore are uncertain and unstable for matching, particularly at the risk of error prop- agation. In this paper, we present an original semantics-oriented attention and deep fusion network (OSOA-DFN) for sentence matching. Unlike existing models, each attention layer of OSOA-DFN is oriented to the original seman- tic representation of another sentence, which captures the relevant information from a fixed matching target. The multiple attention lay- ers allow one sentence to repeatedly read the important information of another sentence for better matching. We then additionally design deep fusion to propagate the attention informa- tion at each matching layer. At last, we in- troduce a self-attention mechanism to capture global context to enhance attention-aware rep- resentation within each sentence. Experiment results on three sentence matching benchmark datasets SNLI, SciTail and Quora show that OSOA-DFN has the ability to model sentence matching more precisely. 1 Introduction Natural language sentence matching is a key tech- nique of comparing two sentences and identifying the semantic relationship between them, which is usually viewed as a classification problem (Wang et al., 2017). The technique has applications in natural language inference to judge whether a hy- pothesis sentence can be inferred from a premise sentence (Bowman et al., 2015) and in paraphrase identification to determine whether two sentences express the equivalent meaning or not (Yin et al., 2015). The core issue for sentence matching is to model the relatedness between two sentences (Rockt¨aschel et al., 2015; Parikh et al., 2016; Wang et al., 2017; Duan et al., 2018). Recently, neural network-based models for sen- tence matching have attracted more attention for their powerful ability to learn sentence represen- tation (Bowman et al., 2015; Wang et al., 2017; Duan et al., 2018). There are mainly two types of frameworks: sentence encoding based frame- work and attention-based framework. For the first type of framework, a simple and effective model is proposed by using two sentence vectors (Bowman et al., 2015), but the interaction between two sen- tences is neglected. For the second type of frame- work, attention mechanism is introduced to model word-level interaction between two sentences and a higher accuracy is achieved (Rockt¨aschel et al., 2015; Parikh et al., 2016; Wang et al., 2017). Particularly, multi-layered deep matching network with attention shows that deeper models outper- form shallower models (Duan et al., 2018). However, the existing attention …

Content truncated to 500 words