Learning Contextual Embeddings for Structural Semantic Similarity using Categorical Information

Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 260–270, Vancouver, Canada, August 3 - August 4, 2017. c⃝2017 Association for Computational Linguistics Learning Contextual Embeddings for Structural Semantic Similarity using Categorical Information Massimo Nicosia♦ and Alessandro Moschitti ♦DISI, University of Trento 38123 Povo (TN), Italy Qatar Computing Research Institute, HBKU, 34110, Doha, Qatar {m.nicosia,amoschitti}@gmail.com Abstract Tree kernels (TKs) and neural networks are two effective approaches for automatic feature engineering. In this paper, we combine them by modeling context word similarity in semantic TKs. This way, the latter can operate subtree matching by applying neural-based similarity on tree lexical nodes. We study how to learn representations for the words in context such that TKs can exploit more focused information. We found that neural em- beddings produced by current methods do not provide a suitable contextual similar- ity. Thus, we define a new approach based on a Siamese Network, which produces word representations while learning a bi- nary text similarity. We set the latter con- sidering examples in the same category as similar. The experiments on question and sentiment classification show that our se- mantic TK highly improves previous re- sults. 1 Introduction Structural Kernels (Moschitti, 2006) can automat- ically represent syntactic and semantic structures in terms of substructures, showing high accuracy in several tasks, e.g., relation extraction (Nguyen et al., 2009; Nguyen and Moschitti, 2011; Plank and Moschitti, 2013; Nguyen et al., 2015) and sen- timent analysis (Nguyen and Shirai, 2015). At the same time, deep learning has demon- strated its effectiveness on a plethora of NLP tasks such as Question Answering (QA) (Severyn and Moschitti, 2015a; Rao et al., 2016), and pars- ing (Andor et al., 2016), to name a few. Deep learning models (DLMs) usually do not include traditional features; they extract relevant signals from distributed representations of words, by ap- plying a sequence of linear and non linear func- tions to the input. Word representations are learned from large corpora, or directly from the training data of the task at hand. Clearly, joining the two approaches above would have the advantage of easily integrating structures with kernels, and lexical representations with embeddings into learning algorithms. In this respect, the Smoothed Partial Tree Kernel (SPTK) is a noticeable approach for using lexical similar- ity in tree structures (Croce et al., 2011). SPTK can match different tree fragments, provided that they only differ in lexical nodes. Although the results were excellent, the used similarity did not consider the fact that words in context assume dif- ferent meanings or weights for the final task, i.e., it does not consider the context. In contrast, SPTK would benefit to use specific word similarity when matching subtrees corresponding to different con- stituency. For example, the two questions: – What famous model was married to Billy Joel? – What famous model of the Universe was pro- posed? are similar in terms of structures and words but clearly have different meaning and also different categories: the first asks for a human …

Content truncated to 500 words