Probing for Semantic Classes: Diagnosing the Meaning Content of Word Embeddings

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5740–5753 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5740 Probing for Semantic Classes: Diagnosing the Meaning Content of Word Embeddings Yadollah Yaghoobzadeh1 Katharina Kann2 Timothy J. Hazen1 Eneko Agirre3 Hinrich Sch¨utze4 1Microsoft Research Montr´eal 2Center for Data Science, New York University 3IXA NLP Group, University of the Basque Country 4CIS, LMU Munich yayaghoo@microsoft.com Abstract Word embeddings typically represent differ- ent meanings of a word in a single conflated vector. Empirical analysis of embeddings of ambiguous words is currently limited by the small size of manually annotated resources and by the fact that word senses are treated as unrelated individual concepts. We present a large dataset based on manual Wikipedia an- notations and word senses, where word senses from different words are related by semantic classes. This is the basis for novel diagnos- tic tests for an embedding’s content: we probe word embeddings for semantic classes and an- alyze the embedding space by classifying em- beddings into semantic classes. Our main find- ings are: (i) Information about a sense is gen- erally represented well in a single-vector em- bedding – if the sense is frequent. (ii) A clas- sifier can accurately predict whether a word is single-sense or multi-sense, based only on its embedding. (iii) Although rare senses are not well represented in single-vector embed- dings, this does not have negative impact on an NLP application whose performance depends on frequent senses. 1 Introduction Word embeddings learned by methods like Word2vec (Mikolov et al., 2013) and Glove (Pen- nington et al., 2014) have had a big impact on natural language processing (NLP) and informa- tion retrieval (IR). They are effective and effi- cient for many tasks. More recently, contextual- ized embeddings like ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018) have further im- proved performance. To understand both word and contextualized embeddings, which still rely on word/subword embeddings at their lowest layer, we must peek inside the blackbox embeddings. Given the importance of word embeddings, at- tempts have been made to construct diagnostic tools to analyze them. However, the main tool for analyzing their semantic content is still look- ing at nearest neighbors of embeddings. Nearest neighbors are based on full-space similarity ne- glecting the multifacetedness property of words (Gladkova and Drozd, 2016) and making them un- stable (Wendlandt et al., 2018). As an alternative, we propose diagnostic clas- sification of embeddings into semantic classes as a probing task to reveal their meaning con- tent. We will refer to semantic classes as S- classes. We use S-classes such as food, drug and living-thing to define word senses. S- classes are frequently used for semantic analysis, e.g., by Kohomban and Lee (2005), Ciaramita and Altun (2006) and Izquierdo et al. (2009) for word sense disambiguation, but have not been used for analyzing embeddings. Analysis based on S-classes is only promising if we have high-quality S-class annotations. Existing datasets are either …

Content truncated to 500 words