Vector of Locally-Aggregated Word Embeddings (VLAWE): A Novel Document-level Representation

Proceedings of NAACL-HLT 2019, pages 363–369 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 363 Vector of Locally-Aggregated Word Embeddings (VLAWE): A Novel Document-level Representation Radu Tudor Ionescu and Andrei M. Butnaru University of Bucharest Department of Computer Science 14 Academiei, Bucharest, Romania raducu.ionescu@gmail.com butnaruandreimadalin@gmail.com Abstract In this paper, we propose a novel representa- tion for text documents based on aggregating word embedding vectors into document em- beddings. Our approach is inspired by the Vec- tor of Locally-Aggregated Descriptors used for image representation, and it works as fol- lows. First, the word embeddings gathered from a collection of documents are clustered by k-means in order to learn a codebook of semnatically-related word embeddings. Each word embedding is then associated to its near- est cluster centroid (codeword). The Vec- tor of Locally-Aggregated Word Embeddings (VLAWE) representation of a document is then computed by accumulating the differ- ences between each codeword vector and each word vector (from the document) associated to the respective codeword. We plug the VLAWE representation, which is learned in an unsuper- vised manner, into a classifier and show that it is useful for a diverse set of text classifi- cation tasks. We compare our approach with a broad range of recent state-of-the-art meth- ods, demonstrating the effectiveness of our ap- proach. Furthermore, we obtain a considerable improvement on the Movie Review data set, reporting an accuracy of 93.3%, which repre- sents an absolute gain of 10% over the state- of-the-art approach. Our code is available at https://github.com/raduionescu/vlawe-boswe/. 1 Introduction In recent years, word embeddings (Bengio et al., 2003; Collobert and Weston, 2008; Mikolov et al., 2013; Pennington et al., 2014) have had a huge impact in natural language processing (NLP) and related fields, being used in many tasks includ- ing sentiment analysis (Dos Santos and Gatti, 2014; Fu et al., 2018), information retrieval (Clin- chant and Perronnin, 2013; Ye et al., 2016) and word sense disambiguation (Bhingardive et al., 2015; Butnaru et al., 2017; Chen et al., 2014; Ia- cobacci et al., 2016), among many others. Start- ing from word embeddings, researchers proposed various ways of aggregating word embedding vec- tors to obtain efficient sentence-level or document- level representations (Butnaru and Ionescu, 2017; Cheng et al., 2018; Clinchant and Perronnin, 2013; Conneau et al., 2017; Cozma et al., 2018; Fu et al., 2018; Hill et al., 2016; Kiros et al., 2015; Kusner et al., 2015; Le and Mikolov, 2014; Shen et al., 2018; Torki, 2018; Zhao et al., 2015; Zhou et al., 2016, 2018). Although the mean (or sum) of word vectors is commonly adopted because of its simplicity (Mitchell and Lapata, 2010), it seems that more complex approaches usually yield better performance (Cheng et al., 2018; Conneau et al., 2017; Cozma et al., 2018; Fu et al., 2018; Hill et al., 2016; Kiros et al., 2015; Torki, 2018; Zhao et al., 2015; Zhou et al., 2016, 2018). To this end, we propose a simple yet effective approach for aggregating word embeddings into …

Content truncated to 500 words