Ranking Kernels for Structures and Embeddings: A Hybrid Preference and Classification Model

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 897–902 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Ranking Kernels for Structures and Embeddings: A Hybrid Preference and Classification Model Kateryna Tymoshenko† and Daniele Bonadiman† and Alessandro Moschitti †DISI, University of Trento, 38123 Povo (TN), Italy Qatar Computing Research Institute, HBKU, 34110, Doha, Qatar {kateryna.tymoshenko,d.bonadiman}@unitn.it amoschitti@gmail.com Abstract Recent work has shown that Tree Ker- nels (TKs) and Convolutional Neural Net- works (CNNs) obtain the state of the art in answer sentence reranking. Additionally, their combination used in Support Vec- tor Machines (SVMs) is promising as it can exploit both the syntactic patterns cap- tured by TKs and the embeddings learned by CNNs. However, the embeddings are constructed according to a classification function, which is not directly exploitable in the preference ranking algorithm of SVMs. In this work, we propose a new hy- brid approach combining preference rank- ing applied to TKs and pointwise rank- ing applied to CNNs. We show that our approach produces better results on two well-known and rather different datasets: WikiQA for answer sentence selection and SemEval cQA for comment selection in Community Question Answering. 1 Introduction Recent work on learning to rank (L2R) has shown that deep learning and kernel methods are two very effective approaches, given their ability of engineering features. In particular, in question answering (QA), Convolutional Neural Networks (CNN), e.g., (Severyn and Moschitti, 2015; Miao et al., 2016; Yin et al., 2016) can automatically learn the representation of question and answer passage (Q/AP) in terms of word embeddings and their non-linear transformations. These are then used by the other layers of the network to mea- sure Q/AP relatedness. In contrast, Convolution Tree Kernels (CTK) can be applied to relational structures built on top of syntactic/semantic struc- tures derived from Q/AP text (Tymoshenko et al., 2016a). CNNs as well as CTKs can achieve the state of the art in ranking APs or also ques- tions. Considering their complementary approach for generating features, studying ways to com- bine them is very promising. In (Tymoshenko et al., 2016a), we investigated the idea of extract- ing layers from CNNs and using them in a ker- nel function to be further combined with CTKs in a composite reranking kernel. This was used in an SVMRank (Joachims, 2002) model, which ob- tained a significant improvement over the individ- ual methods. However, the simple use of CNN layers as vectors in a preference ranking approach is intutively not optimal since such layers are basi- cally learnt in a classification model, thus they are not optimized for SVMRank. In this work, we further compare and investi- gate different ways of combining CTKs and CNNs in reranking settings. In particular, we follow the intuition that as CNNs learn the embeddings in a classification setting they should be used in the same way for building the reranking kernel, i.e., we need to use the embeddings in a pointwise reranking fashion. Therefore, we propose a hy- brid preference-pointwise …

Content truncated to 500 words