Online Learning of Interpretable Word Embeddings

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1687–1692, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Online Learning of Interpretable Word Embeddings Hongyin Luo1, Zhiyuan Liu1,2 ∗, Huanbo Luan1, Maosong Sun1,2 1 Department of Computer Science and Technology, State Key Lab on Intelligent Technology and Systems, National Lab for Information Science and Technology, Tsinghua University, Beijing, China 2 Jiangsu Collaborative Innovation Center for Language Competence, Jiangsu, China Abstract Word embeddings encode semantic mean- ings of words into low-dimension word vectors. In most word embeddings, one cannot interpret the meanings of specific dimensions of those word vectors. Non- negative matrix factorization (NMF) has been proposed to learn interpretable word embeddings via non-negative constraints. However, NMF methods suffer from scale and memory issue because they have to maintain a global matrix for learning. To alleviate this challenge, we propose on- line learning of interpretable word embed- dings from streaming text data. Exper- iments show that our model consistently outperforms the state-of-the-art word em- bedding methods in both representation a- bility and interpretability. The source code of this paper can be obtained from http: //github.com/skTim/OIWE. 1 Introduction Word embeddings (Turian et al., 2010) aim to encode semantic meanings of words into low- dimensional dense vectors. As compared with tra- ditional one-hot representation and distributional representation, word embeddings can better ad- dress the sparsity issue and have achieved success in many NLP applications recent years. There are two typical approaches for word em- beddings. The neural-network (NN) approach (Bengio et al., 2006) employs neural-based tech- niques to learn word embeddings. The matrix fac- torization (MF) approach (Pennington et al., 2014) builds word embeddings by factorizing word- context co-occurrence matrices. The MF approach requires a global statistical matrix, while the N- N approach can flexibly perform learning from ∗Corresponding author: Z. Liu (liuzy@tsinghua.edu.cn) streaming text data, which is efficient in both com- putation and memory. For example, two recen- t NN methods, Skip-Gram and Continuous Bag- of-Word Model (CBOW) (Mikolov et al., 2013a; Mikolov et al., 2013b), have achieved impressive impact due to their simplicity and efficiency. For most word embedding methods, a critical issue is that, we are unaware of what each dimen- sion represent in word embeddings. Hence, the latent dimension for which a word has its largest value is difficult to interpret. This makes word em- beddings like a black-box, and prevents them from being human-readable and further manipulation. People have proposed non-negative matrix fac- torization (NMF) for word representation, denoted as non-negative sparse embedding (NNSE) (Mur- phy et al., 2012). NNSE realizes interpretable word embeddings by applying non-negative con- straints for word embeddings. Although NNSE learns word embeddings with good interpret- abilities, like other MF methods, it also requires a global matrix for learning, thus suffers from heavy memory usage and cannot well deal with stream- ing text data. Inspired by the characteristics of NMF meth- ods (Lee and Seung, 1999), we note that, non- negative constraints only allow additive combi- nations instead …

Content truncated to 500 words