Analytical Methods for Interpretable Ultradense Word Embeddings

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1185–1191, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1185 Analytical Methods for Interpretable Ultradense Word Embeddings Philipp Dufter, Hinrich Sch¨utze Center for Information and Language Processing (CIS) LMU Munich, Germany philipp@cis.lmu.de Abstract Word embeddings are useful for a wide vari- ety of tasks, but they lack interpretability. By rotating word spaces, interpretable dimensions can be identified while preserving the informa- tion contained in the embeddings without any loss. In this work, we investigate three meth- ods for making word spaces interpretable by rotation: Densifier (Rothe et al., 2016), linear SVMs and DensRay, a new method we pro- pose. In contrast to Densifier, DensRay can be computed in closed form, is hyperparameter- free and thus more robust than Densifier. We evaluate the three methods on lexicon induc- tion and set-based word analogy. In addition we provide qualitative insights as to how inter- pretable word spaces can be used for removing gender bias from embeddings. 1 Introduction Distributed representations for words have been of interest in natural language processing for many years. Word embeddings have been particularly effective and successful. On the downside, em- beddings are generally not interpretable. But in- terpretability is desirable for several reasons. i) Semantically or syntactically similar words can be extracted: e.g., for lexicon induction. ii) Interpretable dimensions can be used to evaluate word spaces by examining which information is covered by the embeddings. iii) Computational advantage: for a high-quality sentiment classifier only a couple of dimensions of a high-dimensional word space are relevant. iv) By removing inter- pretable dimensions one can remove unwanted in- formation (e.g., gender bias). v) Most importantly, interpretable embeddings support the goal of inter- pretable deep learning models. Orthogonal transformations have been of par- ticular interest in the literature. The reason is twofold: under the assumption that existing word embeddings are of high-quality one would like to preserve the original embedding structure by using orthogonal transformations (i.e., preserving orig- inal distances). Park et al. (2017) provide evi- dence that rotating existing dense word embed- dings achieves the best performance across a range of interpretability tasks. In this work we modify the objective function of Densifier (Rothe et al., 2016) such that a closed form solution becomes available. We call this method DensRay. Following Amir et al. (2015) we compute simple linear SVMs, which we find to perform surprisingly well. We compare these methods on the task of lexicon induction. Further, we show how interpretable word spaces can be applied to other tasks: first we use inter- pretable word spaces for debiasing embeddings. Second we show how they can be used for solv- ing the set-based word analogy task. To this end, we introduce the set-based method IntCos, which is closely related to LRCos introduced by Drozd et al. (2016). We find IntCos to perform compa- rable to LRCos, but to be preferable for analogies which …

Content truncated to 500 words