Pairwise Word Interaction Modeling with Deep Neural Networks for Semantic Similarity Measurement

Proceedings of NAACL-HLT 2016, pages 937–948, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Pairwise Word Interaction Modeling with Deep Neural Networks for Semantic Similarity Measurement Hua He1 and Jimmy Lin2 1 Department of Computer Science, University of Maryland, College Park 2 David R. Cheriton School of Computer Science, University of Waterloo huah@umd.edu, jimmylin@uwaterloo.ca Abstract Textual similarity measurement is a challeng- ing problem, as it requires understanding the semantics of input sentences. Most previ- ous neural network models use coarse-grained sentence modeling, which has difficulty cap- turing fine-grained word-level information for semantic comparisons. As an alternative, we propose to explicitly model pairwise word in- teractions and present a novel similarity focus mechanism to identify important correspon- dences for better similarity measurement. Our ideas are implemented in a novel neural net- work architecture that demonstrates state-of- the-art accuracy on three SemEval tasks and two answer selection tasks. 1 Introduction Given two pieces of text, measuring their seman- tic textual similarity (STS) remains a fundamental problem in language research and lies at the core of many language processing tasks, including question answering (Lin, 2007), query ranking (Burges et al., 2005), and paraphrase generation (Xu, 2014). Traditional NLP approaches, e.g., developing hand-crafted features, suffer from sparsity because of language ambiguity and the limited amount of annotated data available. Neural networks and dis- tributed representations can alleviate such sparsity, thus neural network-based models are widely used by recent systems for the STS problem (He et al., 2015; Tai et al., 2015; Yin and Sch¨utze, 2015). However, most previous neural network ap- proaches are based on sentence modeling, which first maps each input sentence into a fixed-length vector and then performs comparisons on these representations. Despite its conceptual simplic- ity, researchers have raised concerns about this ap- proach (Mooney, 2014): Will fine-grained word- level information, which is crucial for similarity measurement, get lost in the coarse-grained sen- tence representations? Is it really effective to “cram” whole sentence meanings into fixed-length vectors? In contrast, we focus on capturing fine-grained word-level information directly. Our contribution is twofold: First, instead of using sentence modeling, we propose pairwise word interaction modeling that encourages explicit word context interactions across sentences. This is inspired by our own intuitions of how people recognize textual similarity: given two sentences sent1 and sent2, a careful reader might look for corresponding semantic units, which we op- erationalize in our pairwise word interaction model- ing technique (Sec. 5). Second, based on the pair- wise word interactions, we describe a novel simi- larity focus layer which helps the model selectively identify important word interactions depending on their importance for similarity measurement. Since not all words are created equal, important words that can make more contributions deserve extra “focus” from the model (Sec. 6). We conducted thorough evaluations on ten test sets from three SemEval STS competitions (Agirre et al., 2012; Marelli et al., 2014; Agirre et al., 2014) and two answer selection tasks (Yang et al., 2015; Wang et al., 2007). We outperform the …

Content truncated to 500 words