Explaining Word Embeddings via Disentangled Representation

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 720–725 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 720 Explaining Word Embeddings via Disentangled Representation Keng-Te Liao National Taiwan University d05922001@ntu.edu.tw Cheng-Syuan Lee National Taiwan University r07922055@ntu.edu.tw Zhong-Yu Huang National Taiwan University r06944047@ntu.edu.tw Shou-de Lin National Taiwan University sdlin@csie.ntu.edu.tw Abstract Disentangled representations have attracted in- creasing attention recently. However, how to transfer the desired properties of disentangle- ment to word representations is unclear. In this work, we propose to transform typical dense word vectors into disentangled embeddings featuring improved interpretability via encod- ing polysemous semantics separately. We also found the modular structure of our disentan- gled word embeddings helps generate more ef- ficient and effective features for natural lan- guage processing tasks. 1 Introduction Disentangled representations are known to repre- sent interpretable factors in separated dimensions. This property can potentially help people under- stand or discover knowledge in the embeddings. In natural language processing (NLP), works of dis- entangled representations have shown notable im- pacts on sentence and document-level applications. For example, Larsson et al. (2017) and Melnyk et al. (2017) proposed to disentangle sentiment and semantic of sentences. By manipulating sentiment factors, the machine can rewrite a sentence with dif- ferent sentiment. Brunner et al. (2018) also demon- strated sentence generation while more focusing on syntactic factors such as part-of-speech tags. For document-level applications, Jain et al. (2018) pre- sented a learning algorithm which embeds biomed- ical abstracts disentangling populations, interven- tions and outcomes. Regarding word-level disen- tanglement, Athiwaratkun and Wilson (2017) pro- posed mixture of Gaussian models which can dis- entangle meanings of polysemous words into two or three clusters. It has a connection with unsu- pervised sense representations (Camacho-Collados and Pilehvar, 2018) which is an active research topic in the community. In this work, we focus on word-level disentangle- ment and introduce an idea of transforming dense word embeddings such as GloVe (Pennington et al., 2014) or word2vec (Mikolov et al., 2013b) into disentangled word embeddings (DWE). The main feature of our DWE is that it can be segmented into multiple sub-embeddings or sub-areas as il- lustrated in Figure 1. In the figure, each sub-area encodes information relevant to one specific topical factor such as Animal or Location. As an example, we found words similar to “turkey” are “geese”, “flock” and “goose” in the Animal area, and the similar words turn into “Greece”, “Cyprus” and “Ankara” in the Location area. Figure 1: Disentangled embedding with factors Animal, Location and Unseen. We also found our DWE generally satisfies the Modularity and Compactness properties proposed by Higgins et al. (2018) and Ridgeway and Mozer (2018) which can be a definition of general-purpose disentangled representations. Also, our DWE can have the following advantages: • Explaining Underlying Knowledge The multi-senses of words can be extracted and separately encoded despite the learning algorithm of the original word embeddings (e.g. GloVe) does not do …

Content truncated to 500 words