Investigating Word-Class Distributions in Word Vector Spaces

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3657–3666 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3657 Investigating Word-Class Distributions in Word Vector Spaces Ryohei Sasano Graduate School of Informatics Nagoya University sasano@i.nagoya-u.ac.jp Anna Korhonen Language Technology Lab University of Cambridge alk23@cam.ac.uk Abstract This paper presents an investigation on the dis- tribution of word vectors belonging to a cer- tain word class in a pre-trained word vector space. To this end, we made several assump- tions about the distribution, modeled the distri- bution accordingly, and validated each assump- tion by comparing the goodness of each model. Specifically, we considered two types of word classes – the semantic class of direct objects of a verb and the semantic class in a thesaurus – and tried to build models that properly esti- mate how likely it is that a word in the vector space is a member of a given word class. Our results on selectional preference and WordNet datasets show that the centroid-based model will fail to achieve good enough performance, the geometry of the distribution and the exis- tence of subgroups will have limited impact, and also the negative instances need to be con- sidered for adequate modeling of the distribu- tion. We further investigated the relationship between the scores calculated by each model and the degree of membership and found that discriminative learning-based models are best in finding the boundaries of a class, while mod- els based on the offset between positive and negative instances perform best in determining the degree of membership. 1 Introduction Several studies have been successful in represent- ing the meaning of a word with a vector in a con- tinuous vector space (e.g., Mikolov et al. 2013a; Pennington et al. 2014). These representations are useful for a range of natural language process- ing (NLP) tasks. The interpretation and geometry of the word embeddings have also attracted atten- tion (e.g., Kim and de Marneffe 2013; Mimno and Thompson 2017). However, little attention has been paid to the distribution of words belonging to a certain word class in a word vector space, though Centroid of negative instances Centroid of positive instances Figure 1: 2D t-SNE projection of GloVe vectors. The 200 plus symbols (+) represent the word vectors that can be a direct object of the verb play (positive in- stances) and the 1000 squares ( ■) represent other word vectors (negative instances). empirical analysis of such a distribution provides a better understanding of word vector spaces and insight into algorithmic choices for several NLP tasks, including selectional preference acquisition and entity set expansion. Figure 1 shows a 2D projection of word em- beddings. We extracted 200 words that can be a direct object of the verb play (positive instances) and 1000 other words (negative instances) and pro- jected their GloVe vectors (Pennington et al., 2014) into two dimensions using t-distributed Stochastic Neighbor Embedding (t-SNE) (van der Maaten and Hinton, 2008). The plus symbols (+) represent the positive …

Content truncated to 500 words