Rotated Word Vector Representations and their Interpretability

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 401–411 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Rotated Word Vector Representations and their Interpretability Sungjoon Park and JinYeong Bak and Alice Oh Department of Computing, KAIST, Republic of Korea {sungjoon.park, jy.bak}@kaist.ac.kr, alice.oh@kaist.edu Abstract Vector representation of words improves performance in various NLP tasks, but the high-dimensional word vectors are very difficult to interpret. We apply several ro- tation algorithms to the vector representa- tion of words to improve the interpretabil- ity. Unlike previous approaches that in- duce sparsity, the rotated vectors are in- terpretable while preserving the expressive performance of the original vectors. Fur- thermore, any pre-built word vector repre- sentation can be rotated for improved in- terpretability. We apply rotation to skip- grams and glove and compare the expres- sive power and interpretability with the original vectors and the sparse overcom- plete vectors. The results show that the ro- tated vectors outperform the original and the sparse overcomplete vectors for inter- pretability and expressiveness tasks. 1 Introduction Vector representations of words contain rich se- mantic and syntactic information and thus improve the performance of numerous natural language processing tasks. The vectors also play a basic role as an embedding layer in deep learning models for NLP, affecting the expressive performance of the model (Iyyer et al., 2014; Tai et al., 2015; Yang et al., 2016). However, the many dimensions com- prising the vector representation are not amenable to interpretation. Previous research on vector representation of words has proposed improving interpretability while keeping the expressive performance by in- ducing sparsity in word vector dimensions (Mur- phy et al., 2012; Fyshe et al., 2014). Recent re- search has proposed to build sparse vector repre- sentations from a large corpus and added the non- negativity constraint using improved projected gradient (Luo et al., 2015), while (Sun et al., 2016) learns l1-regularised vectors. But, these models cannot be learned over pre-trained word vectors based on skip-gram (Mikolov et al., 2013) or glove (Pennington et al., 2014) which are widely used. Faruqui et al. proposes an alternative approach to stand-alone models by forming sparse representa- tions based on the pre-trained models. To do this, they use overcomplete vectors, which are much higher in dimensionality than the original vectors. Unlike these sparsity-inducing approaches, we construct an interpretable word vector representa- tion by using the pre-trained word vectors as in- put and using a basis rotation algorithm from the Exploratory Factor Analysis (EFA) literature used in developing psychological scales (Osborne and Costello, 2009). Like the word vector representa- tion, every single item in the scale is represented as a numeric vector in the latent factor space. The set of item vectors are represented in a factor loading matrix, and the matrix is rotated such that the fac- tors (i.e., dimensions) become interpretable. The rotation achieves a Simple Structure (Thurstone, 1947) through minimizing the row and the column complexity of the matrix (Crawford and Ferguson, 1970). We elaborate on this …

Content truncated to 500 words