Unsupervised Sparse Vector Densification for Short Text Similarity

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 1275–1280, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Unsupervised Sparse Vector Densification for Short Text Similarity Yangqiu Song and Dan Roth Department of Computer Science University of Illinois at Urbana-Champaign Urbana, IL 61801, USA {yqsong,danr}@illinois.edu Abstract Sparse representations of text such as bag-of- words models or extended explicit semantic analysis (ESA) representations are commonly used in many NLP applications. However, for short texts, the similarity between two such s- parse vectors is not accurate due to the small term overlap. While there have been multiple proposals for dense representations of words, measuring similarity between short texts (sen- tences, snippets, paragraphs) requires combin- ing these token level similarities. In this paper, we propose to combine ESA representations and word2vec representations as a way to gen- erate denser representations and, consequent- ly, a better similarity measure between short texts. We study three densification mecha- nisms that involve aligning sparse representa- tion via many-to-many, many-to-one, and one- to-one mappings. We then show the effective- ness of these mechanisms on measuring simi- larity between short texts. 1 Introduction Bag-of-words model has been used for many ap- plications as the state-of-the-art method for tasks such as document classifications and information re- trieval. It represents each text as a bag-of-words, and computes the similarity, e.g., cosine value, be- tween two sparse vectors in the high-dimensional space. When the contextual information is insuffi- cient, e.g., due to the short length of the document, explicit semantic analysis (ESA) has been used as a way to enrich the text representation (Gabrilovich and Markovitch, 2006; Gabrilovich and Markovitch, 2007). Instead of using only the words in a doc- ument, ESA uses a bag-of-concepts retrieved from Wikipedia to represent the text. Then the similarity between two texts can be computed in this enriched concept space. Both bag-of-words and bag-of-concepts model- s suffer from the sparsity problem. Because both models use sparse vectors to represent text, when comparing two pieces of texts, the similarity can be zero even when the text snippets are highly related, but make use of different vocabulary. We can expect that these two texts are related but the similarity val- ue does not reflect that. ESA, despite augmenting the lexical space with relevant Wikipedia concepts, still suffers from the sparsity problem. We illustrate this problem with the following simple experiment, done by choosing a documents from the “rec.autos” group in the 20-newsgroups data set1. For both doc- uments and the label description “cars” (here we fol- low the description shown in (Chang et al., 2008; Song and Roth, 2014)), we computed 500 concepts using ESA. Then we identified the concepts that ap- pear both in the document ESA representation and in the label ESA representation. The average sizes of this intersection (number of overlapping concepts in the document and label representation) are shown in Table 1. In addition to the original documents, we also split each …

Content truncated to 500 words