Convolutional Sentence Kernel from Word Embeddings for Short Text Categorization

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 775–780, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Convolutional Sentence Kernel from Word Embeddings for Short Text Categorization Jonghoon Kim François Rousseau LIX, École Polytechnique, France john.jonghoon.kim@gmail.com Michalis Vazirgiannis Abstract This paper introduces a convolutional sen- tence kernel based on word embeddings. Our kernel overcomes the sparsity issue that arises when classifying short docu- ments or in case of little training data. Ex- periments on six sentence datasets showed statistically significant higher accuracy over the standard linear kernel with n- gram features and other proposed models. 1 Introduction With the proliferation of text data available on- line, text categorization emerged as a prominent research topic. Traditionally, words (unigrams) and phrases (n-grams) have been considered as document features and subsequently fed to a clas- sifier such as an SVM (Joachims, 1998). In the SVM dual formulation that relies on kernels, i. e. similarity measures between documents, a linear kernel can be interpreted as the number of ex- act matching n-grams between two documents. Consequently, for short documents or when lit- tle training data is available, sparsity issues due to word synonymy arise, e. g., the sentences ‘John likes hot beverages’ and ‘John loves warm drinks’ have little overlap and therefore low linear kernel value (only 1) in the n-gram feature space, even with dependency tree representations and down- ward paths for n-grams as illustrated in Figure 1. We propose to relax the exact matching between words by capitalizing on distances in word embed- dings. We smooth the implicit delta word kernel, i. e. a Dirac similarity function between unigrams, behind the traditional linear document kernel to capture the similarity between words that are dif- ferent, yet semantically close. We then aggregate these word and phrase kernels into sentence and documents kernels through convolution resulting in higher kernel values between semantically re- lated sentences (e. g., close to 7 compared to 1 John likes beverages hot (a) ‘John likes hot beverages’ ≈ John loves drinks warm (b) ‘John loves warm drinks’ Figure 1: Dependency tree representations of se- mantically related sentences yet with little overlap. with bigram downward paths in Figure 1). Ex- periments on six standard datasets for sentiment analysis, subjectivity detection and topic spotting showed statistically significant higher accuracy for our proposed kernel over the bigram approaches. Our main goal is to demonstrate empirically that word distances from a given word vector space can easily be incorporated in the standard kernel be- tween documents for higher effectiveness and lit- tle additional cost in efficiency. The rest of this paper is structured as follows. Section 2 reviews the related work. Section 3 gives the detailed formulation of our kernel. Section 4 describes the experimental settings and the results we obtained on several datasets. Finally, Section 5 concludes our paper and mentions future work. 2 Related work Siolas and d’Alché Buc (2000) pioneered the idea of semantic kernels for text categorization, cap- italizing on WordNet (Miller, …

Content truncated to 500 words