Interpretable Word Embeddings via Informative Priors

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 6323–6329, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 6323 Interpretable Word Embeddings via Informative Priors Miriam Hurtado Bodell∗ Link¨oping University miriam.hurtado.bodell@liu.se Martin Arvidsson∗ Link¨oping University martin.arvidsson@liu.se M˚ans Magnusson Aalto University mans.magnusson@aalto.fi Abstract Word embeddings have demonstrated strong performance on NLP tasks. However, lack of interpretability and the unsupervised na- ture of word embeddings have limited their use within computational social science and digital humanities. We propose the use of informative priors to create interpretable and domain-informed dimensions for probabilistic word embeddings. Experimental results show that sensible priors can capture latent semantic concepts better than or on-par with the current state of the art, while retaining the simplicity and generalizability of using priors. 1 Introduction Increased availability of large digitized corpora and significant developments in natural language processing (NLP) have sparked a growing inter- est within computational social science and digital humanities (CSSDH) to use computational meth- ods for textual data (Laver et al., 2003; Grimmer, 2010; DiMaggio et al., 2013; Jockers and Mimno, 2013; Tsur et al., 2015). Word embeddings, a family of unsupervised methods for representing words as dense vectors (Mikolov et al., 2013b; Pennington et al., 2014), are one such develop- ment. Although word embeddings have demon- strated strong performance on NLP tasks (Mikolov et al., 2013a,c), they have yet to gain widespread attention within CSSDH. We believe two key limitations can help ex- plain the lack of applications within CSSDH. First, since the dimensions of the word embeddings are largely uninterpretable, it is not clear how to dis- entangle why words are similar. Substantive inter- pretability is key for CSSDH research, and thus, the lack thereof is a major limitation. Second, off- the-shelf word embedding models generally lack a channel through which substantive research ques- tions can be incorporated. ∗ Equal contribution To improve interpretability, previous research suggests using sparsity constraints (Murphy et al., 2012; Sun et al., 2016; Faruqui et al., 2015) and rotation techniques (Park et al., 2017; Rothe and Sch¨utze, 2016; Dufter and Sch¨utze, 2019). Other work considers dimension-specific constraints to remove gender-bias, and, as a by-product, im- prove interpretability (Zhao et al., 2018). Within CSSDH, previous work derives interpretable di- mensions via post-processing in the form of antonym-pair vector algebra (Kozlowski et al., 2018; Garg et al., 2018) and ideal-point anchoring of antonym word-pairs (Lauretig, 2019). Recent formulations of word embeddings as probabilis- tic models (Vilnis and McCallum, 2015; Rudolph et al., 2016; Barkan, 2017; Havrylov et al., 2018) enable the incorporation of domain knowledge through priors. In this paper, we add to the liter- ature on interpretable word embeddings, propos- ing a novel use of informative priors to create pre- defined interpretable dimensions – thus leverag- ing the expressiveness and generalizeability of the probabilistic framework. 2 Informative Priors in Word Embeddings The central idea of this paper is to use informa- tive priors to restrict …

Content truncated to 500 words