Firearms and Tigers are Dangerous, Kitchen Knives and Zebras are Not: Testing whether Word Embeddings Can Tell

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 276 Firearms and Tigers are Dangerous, Kitchen Knives and Zebras are Not: Testing whether Word Embeddings Can Tell Pia Sommerauer and Antske Fokkens Computational Lexicology and Terminology Lab Vrije Universiteit Amsterdam De Boelelaan 1105 Amsterdam, The Netherlands pia.sommerauer@vu.nl, antske.fokkens@vu.nl Abstract This paper presents an approach for inves- tigating the nature of semantic information captured by word embeddings. We propose a method that extends an existing human- elicited semantic property dataset with gold negative examples using crowd judgments. Our experimental approach tests the ability of supervised classifiers to identify semantic fea- tures in word embedding vectors and com- pares this to a feature-identification method based on full vector cosine similarity. The idea behind this method is that properties identi- fied by classifiers, but not through full vec- tor comparison are captured by embeddings. Properties that cannot be identified by either method are not. Our results provide an initial indication that semantic properties relevant for the way entities interact (e.g. dangerous) are captured, while perceptual information (e.g. colors) is not represented. We conclude that, though preliminary, these results show that our method is suitable for identifying which prop- erties are captured by embeddings. 1 Introduction Word embeddings are widely used in NLP and have been shown to boost performance in a large selection of tasks ranging from morphological analysis to sentiment analysis (Lazaridou et al., 2013; Socher et al., 2013; Zhou and Xu, 2015, among many others). Despite a number of dif- ferent approaches to evaluation, our understand- ing of what type of information is represented by the vectors remains limited. Most approaches fo- cus on full-vector comparison which treat vectors as points in a space (Yaghoobzadeh and Sch¨utze, 2016), which are evaluated by performance on semantic similarity or relatedness test sets and analogy questions (Mikolov et al., 2013; Turney, 2012). Previous work, however, has shown that high performance does not necessarily mean that vectors actually contain the information required to solve the task (Rogers et al., 2017; Linzen, 2016). Better understanding of the kind of seman- tic information captured by word embeddings can increase our understanding of how they help im- prove downstream tasks. In general, understand- ing what information is present in (often promi- nent) input embeddings forms an essential com- ponent of gaining deeper understanding of the na- ture of information and manner in which it travels through the hidden layers of a neural network. In this paper, we propose a method that inves- tigates what kind of semantic information is en- coded in vectors using a human-elicited dataset of semantic properties. We compare the output of supervised classifiers to an approach based on full-vector comparison that cannot access individ- ual dimensions. The assumptions behind this ap- proach are that (1) both full-vector comparison and the supervised classifier will perform well on identifying semantic properties that correlate highly with general similarity; (2) the …

Content truncated to 500 words