Incorporating Contextual and Syntactic Structures Improves Semantic Similarity Modeling

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1204–1209, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1204 Incorporating Contextual and Syntactic Structures Improves Semantic Similarity Modeling Linqing Liu,1 Wei Yang,1 Jinfeng Rao,2 Raphael Tang,1 and Jimmy Lin1 1 David R. Cheriton School of Computer Science, University of Waterloo 2 Department of Computer Science, University of Maryland, College Park {linqing.liu,w85yang,r33tang, jimmylin}@uwaterloo.ca, raojinfeng@gmail.com Abstract Semantic similarity modeling is central to many NLP problems such as natural language inference and question answering. Syntactic structures interact closely with semantics in learning compositional representations and al- leviating long-range dependency issues. How- ever, such structure priors have not been well exploited in previous work for semantic mod- eling. To examine their effectiveness, we start with the Pairwise Word Interaction Model, one of the best models according to a recent reproducibility study, then introduce compo- nents for modeling context and structure us- ing multi-layer BiLSTMs and TreeLSTMs. In addition, we introduce residual connections to the deep convolutional neural network com- ponent of the model. Extensive evaluations on eight benchmark datasets show that in- corporating structural information contributes to consistent improvements over strong base- lines. 1 Introduction Modeling the semantic similarity between a pair of sentences is a fundamental task in natural lan- guage processing. It is the core problem of many tasks such as question answering (He et al., 2015; Rao et al., 2017; Wang et al., 2018) and query ranking (Mitra and Craswell, 2019). Re- cently, various neural networks have been pro- posed for textual similarity modeling. These mod- els share three main components: (1) sequential sentence encoders, which incorporate word con- text and sentence order for better sentence rep- resentations, e.g., by using recurrent neural net- works (RNNs; Mikolov et al., 2010; Seo et al., 2016), (2) interaction and attention mechanisms, which use the encoding outputs of sentences to calculate or reweight salient word pair interac- tions (He and Lin, 2016; Chen et al., 2017), and (3) incorporating syntactic parsing information as an intuitive structure prior for sentence modeling (Chen and Manning, 2014; Zhao et al., 2016; Chen et al., 2017). Our work is inspired by the recent reproducibil- ity study by Lan and Xu (2018), which examines many neural network architectures for semantic similarity modeling through extensive evaluations on multiple benchmark datasets. Their results sug- gest that syntactic structure information captured by a TreeLSTM encoder either provides few ben- efits or even hurts performance. Structure infor- mation has often been overlooked in recent seman- tic modeling methods, such as InferSent (Conneau et al., 2017), DecAtt (Parikh et al., 2016), and BiMPM (Wang et al., 2017). It is not yet clear whether the syntactic structures implicitly cap- tured by sequential modeling of texts from large annotated data or existing structure modeling tech- niques (Tai et al., 2015; Kipf and Welling, 2017) are effective in learning tree representations. To further explore the effects of tree structures in …

Content truncated to 500 words