Modelling Salient Features as Directions in Fine-Tuned Semantic Spaces

Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018), pages 530–540 Brussels, Belgium, October 31 - November 1, 2018. c⃝2018 Association for Computational Linguistics 530 Modelling Salient Features as Directions in Fine-Tuned Semantic Spaces Thomas Ager School of CS & Informatics Cardiff University, UK agert@cardiff.ac.uk Ondˇrej Kuˇzelka Dept. of Computer Science KU Leuven, Belgium ondrej.kuzelka@kuleuven.be Steven Schockaert School of CS & Informatics Cardiff University, UK schockaerts1@cardiff.ac.uk Abstract In this paper we consider semantic spaces con- sisting of objects from some particular domain (e.g. IMDB movie reviews). Various authors have observed that such semantic spaces of- ten model salient features (e.g. how scary a movie is) as directions. These feature direc- tions allow us to rank objects according to how much they have the corresponding fea- ture, and can thus play an important role in interpretable classifiers, recommendation sys- tems, or entity-oriented search engines, among others. Methods for learning semantic spaces, however, are mostly aimed at modelling simi- larity. In this paper, we argue that there is an inherent trade-off between capturing similar- ity and faithfully modelling features as direc- tions. Following this observation, we propose a simple method to fine-tune existing seman- tic spaces, with the aim of improving the qual- ity of their feature directions. Crucially, our method is fully unsupervised, requiring only a bag-of-words representation of the objects as input. 1 Introduction Vector space representations, or ‘embeddings’, play a crucial role in various areas of natural lan- guage processing. For instance, word embeddings (Mikolov et al., 2013; Pennington et al., 2014) are routinely used as representations of word mean- ing, while knowledge graph embeddings (Bordes et al., 2013) are used to find plausible missing in- formation in structured knowledge bases, or to ex- ploit such knowledge bases in neural network ar- chitectures. In this paper we focus on domain- specific semantic spaces, i.e. vector space repre- sentations of the objects of a single domain, as opposed to the more heterogeneous setting of e.g. word embeddings. Such domain-specific semantic spaces are used, for instance, to represent items in recommender systems (Vasile et al., 2016; Liang et al., 2016; Van Gysel et al., 2016), to represent entities in semantic search engines (Jameel et al., 2017; Van Gysel et al., 2017), or to represent examples in classification tasks (Demirel et al., 2017). In many semantic spaces it is possible to find directions that correspond to salient features from the considered domain. For instance, Gupta et al. (2015) found that features of countries, such as their GDP, fer- tility rate or even level of CO2 emissions, can be predicted from word embeddings using a linear re- gression model. Similarly, in (Kim and de Marn- effe, 2013) directions in word embeddings were found that correspond to adjectival scales (e.g. bad < okay < good < excellent) while Rothe and Sch¨utze (2016) found directions indicating lexical features such as the frequency of occurrence and polarity of words. Finally, Derrac and Schockaert (2015) found directions corresponding to proper- ties such as ‘Scary’, …

Content truncated to 500 words