Learning and Evaluating Sparse Interpretable Sentence Embeddings

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 200–210 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 200 Learning and Evaluating Sparse Interpretable Sentence Embeddings Valentin Trifonov ETH Z¨urich, Switzerland valentin.trifonov@outlook.com Octavian-Eugen Ganea ETH Z¨urich, Switzerland octavian.ganea@inf.ethz.ch Anna Potapenko National Research University Higher School of Economics, Russia∗ anna.a.potapenko@gmail.com Thomas Hofmann ETH Z¨urich, Switzerland thomas.hofmann@inf.ethz.ch Abstract Previous research on word embeddings has shown that sparse representations, which can be either learned on top of existing dense embeddings or obtained through model con- straints during training time, have the bene- fit of increased interpretability properties: to some degree, each dimension can be under- stood by a human and associated with a rec- ognizable feature in the data. In this paper, we transfer this idea to sentence embeddings and explore several approaches to obtain a sparse representation. We further introduce a novel, quantitative and automated evaluation metric for sentence embedding interpretability, based on topic coherence methods. We observe an increase in interpretability compared to dense models, on a dataset of movie dialogs and on the scene descriptions from the MS COCO dataset. 1 Introduction In the word embeddings literature, it has previ- ously been of interest to find interpretable repre- sentations: individual dimensions should capture a distinct semantic meaning, such that humans are able to understand why a word is encoded in a par- ticular vector. With a cognitive plausibility argu- ment from Murphy et al. (2012), the interpretabil- ity can be linked to sparse representations: they argue that the representation should model a wide range of features in the data and that every sample should be characterized by the presence of a small number of key features. Arora et al. (2016) use this idea to recover and disentangle the different meanings of polysemous words. The above-named approaches, as well as those by Subramanian et al. (2017); Faruqui et al. (2015), recover an interpretable sparse represen- tation in a separate, post-processing step on top of ∗Work done during an internship at ETH Z¨urich. the uninterpretable, dense embeddings of the orig- inal model (often word2vec or GloVe). This is commonly done using sparse coding or a down- stream model. Additionally to understanding a model’s intermediate representation, there has been work on constructing models that inherently use a sparse embedded representation by learning it during the training process (Sun et al., 2016; Chen et al., 2017). This is motivated by the idea that the model should include the prior that each word is a sparse combination of disentan- gled features from the very beginning. In contrast, when computing dense embeddings first, it is less likely that this representation will be easily disen- tanglable in the post-processing step. Goh (2016) argues that sparse representations can be used to explain image and sentence embed- dings as well. To be precise, the author focuses on encoder-decoder neural networks and uses sparse coding to recover interpretable features in the la- tent spaces of a variational autoencoder (Kingma and Welling, 2013) …

Content truncated to 500 words