Vector of Locally Aggregated Embeddings for Text Representation

Proceedings of NAACL-HLT 2019, pages 1408–1414 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1408 Vector of Locally Aggregated Embeddings for Text Representation Hadi Amiria and Mitra Mohtaramib aHarvard, Department of Biomedical Informatics, Boston, MA, USA bMIT, Computer Science and Artificial Intelligence Laboratory, Cambridge, MA, USA hadi@hms.harvard.edu, mitra@csail.mit.edu Abstract We present Vector of Locally Aggregated Embeddings (VLAE) for effective and, ulti- mately, lossless representation of textual con- tent. Our model encodes each input text by effectively identifying and integrating the rep- resentations of its semantically-relevant parts. The proposed model generates high quality representation of textual content and improves the classification performance of current state- of-the-art deep averaging networks across sev- eral text classification tasks. 1 Introduction Representation learning algorithms can reveal in- trinsic low-dimensional structure in data (Rumel- hart et al., 1986; Bengio et al., 2013; LeCun et al., 2015). In particular, deep averaging networks (DANs) are effective for text classification (Shen et al., 2018; Arora et al., 2017; Wieting et al., 2016; Iyyer et al., 2015). They achieve their improvement through use of word embeddings, weighted averaging, and deepening networks. The above works show that DANs can outperform RNNs and CNNs in text classification while tak- ing only a fraction of their training time. In this work, with a special focus on DANs, we study the effect of information loss associated with average word embeddings and develop al- gorithms that are robust against information loss for text representation. We show that divergence of word embeddings from their average can be considered as a good proxy to quantify informa- tion loss; in particular, longer documents suf- fer from significant information loss when repre- sented by average word embeddings. These results inspire our work to develop a novel representa- tion learning approach based on Vector of Locally Aggregated Descriptors (VLAD) (J´egou et al., 2010; Arandjelovic and Zisserman, 2013)–an ef- fective approach to integrate image descriptors for large scale image datasets. Our model identifies semantically-relevant parts of documents and lo- cally integrates their representations through clus- tering and autoencoding. In contrast to averaging, our model prevents larger semantically-relevant parts of inputs to dominate final representations. It improves DANs by 5.30 macro-F1 points in clas- sifying longer texts and show comparable perfor- mance to them on shorter text. 2 Preliminary Analysis How can information loss be quantified when word embeddings are averaged? How important it is to address information loss when represent- ing textual content? Are representation learn- ing algorithms robust against information loss? We conduct experiments to answer these ques- tions with respect to deep averaging network (DANs). Our study can inspire works in more complex averaging approaches such as those re- ported in (Torabi Asr et al., 2018; Kiela et al., 2015) as well as recent works on unsupervised se- mantic similarity (Pagliardini et al., 2018). We use the DAN developed in (Joulin et al., 2017) and several datasets containing short and long docu- ments to answer these questions. 2.1 Quantifying Information Loss Let’s assume a …

Content truncated to 500 words