Ultradense Word Embeddings by Orthogonal Transformation

Proceedings of NAACL-HLT 2016, pages 767–777, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Ultradense Word Embeddings by Orthogonal Transformation Sascha Rothe and Sebastian Ebert and Hinrich Sch¨utze Center for Information and Language Processing LMU Munich, Germany {sascha|ebert}@cis.lmu.de Abstract Embeddings are generic representations that are useful for many NLP tasks. In this paper, we introduce DENSIFIER, a method that learns an orthogonal transformation of the embed- ding space that focuses the information rele- vant for a task in an ultradense subspace of a dimensionality that is smaller by a factor of 100 than the original space. We show that ultradense embeddings generated by DENSI- FIER reach state of the art on a lexicon creation task in which words are annotated with three types of lexical information – sentiment, con- creteness and frequency. On the SemEval2015 10B sentiment analysis task we show that no information is lost when the ultradense sub- space is used, but training is an order of mag- nitude more efficient due to the compactness of the ultradense space. 1 Introduction Embeddings are useful for many tasks, including word similarity (e.g., Pennington et al. (2014)), named entity recognition (NER) (e.g., Collobert et al. (2011)) and sentiment analysis (e.g., Kim (2014), Kalchbrenner et al. (2014), Severyn and Moschitti (2015)). Embeddings are generic representations, containing different types of information about a word. Statistical models can be trained to make best use of these generic representations for a specific ap- plication like NER or sentiment analysis (Ebert et al., 2015). Our hypothesis in this paper is that the informa- tion useful for any given task is contained in an ul- tradense subspace Eu. We propose the new method DENSIFIER to identify Eu. Given a set of word em- beddings, DENSIFIER learns an orthogonal transfor- mation of the original space Eo on a task-specific training set. The orthogonality of the transformation can be considered a hard regularizer. The benefit of the proposed method is that embed- dings are most useful if learned on unlabeled cor- pora and performance-enhanced on a broad array of tasks. This means we should try to keep all informa- tion offered by them. Orthogonal transformations “reorder” the space without adding or removing in- formation and preserve the bilinear form, i.e., Eu- clidean distance and cosine. The transformed em- beddings concentrate all information relevant for the task in Eu. The benefits of Eu compared to Eo are (i) high- quality and (ii) efficient representations. (i) DENSI- FIER moves non-task-related information outside of Eu, i.e., into the orthogonal complement of Eu. As a result, Eu provides higher-quality representations for the task than Eo; e.g., noise that could result in overfitting is reduced in Eu compared to Eo. (ii) Eu has a dimensionality smaller by a factor of 100 in our experiments. As a result, training statistical models on these embeddings is much faster. These models also have many fewer parameters, thus again helping to prevent overfitting, especially for complex, deep neural networks. We show the benefits of …

Content truncated to 500 words