SCDV : Sparse Composite Document Vectors using soft clustering over distributional representations

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 659–669 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics SCDV : Sparse Composite Document Vectors using soft clustering over distributional representations Dheeraj Mekala* IIT Kanpur dheerajm@iitk.ac.in Vivek Gupta* Microsoft Research t-vigu@microsoft.com Bhargavi Paranjape Microsoft Research t-bhpara@microsoft.com Harish Karnick IIT Kanpur hk@iitk.ac.in Abstract We present a feature vector forma- tion technique for documents - Sparse Composite Document Vector (SCDV) - which overcomes several shortcomings of the current distributional paragraph vec- tor representations that are widely used for text representation. In SCDV, word em- beddings are clustered to capture multiple semantic contexts in which words occur. They are then chained together to form document topic-vectors that can express complex, multi-topic documents. Through extensive experiments on multi-class and multi-label classification tasks, we outper- form the previous state-of-the-art method, NTSG (Liu et al., 2015a). We also show that SCDV embeddings perform well on heterogeneous tasks like Topic Coherence, context-sensitive Learning and Informa- tion Retrieval. Moreover, we achieve sig- nificant reduction in training and predic- tion times compared to other representa- tion methods. SCDV achieves best of both worlds - better performance with lower time and space complexity. 1 Introduction Distributed word embeddings represent words as dense, low-dimensional and real-valued vectors that can capture their semantic and syntactic prop- erties. These embeddings are used abundantly by machine learning algorithms in tasks such as text classification and clustering. Traditional bag- of-word models that represent words as indices into a vocabulary don’t account for word ordering and long-distance semantic relations. Represen- tations based on neural network language models *Represents equal contribution (Mikolov et al., 2013b) can overcome these flaws and further reduce the dimensionality of the vec- tors. The success of the method is recently math- ematically explained using the random walk on discourses model (Arora et al., 2016a). However, there is a need to extend word embeddings to en- tire paragraphs and documents for tasks such as document and short-text classification. Representing entire documents in a dense, low- dimensional space is a challenge. A simple weighted average of the word embeddings in a large chunk of text ignores word ordering, while a parse tree based combination of embeddings (Socher et al., 2013) can only extend to sentences. (Le and Mikolov, 2014) trains word and para- graph vectors to predict context but shares word- embeddings across paragraphs. However, words can have different semantic meanings in different contexts. Hence, vectors of two documents that contain the same word in two distinct senses need to account for this distinction for an accurate se- mantic representation of the documents. (Ling et al., 2015), (Liu et al., 2015a) map word em- beddings to a latent topic space to capture differ- ent senses in which words occur. However, they represent complex documents in the same space as words, reducing their expressive power. These methods are also computationally intensive. In this work, we propose the Sparse Compos- ite Document Vector(SCDV) representation learn- ing technique to address these challenges …

Content truncated to 500 words