Improving Distributional Semantic Vectors through Context Selection and Normalisation

Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 230–238, Gothenburg, Sweden, April 26-30 2014. c⃝2014 Association for Computational Linguistics Improving Distributional Semantic Vectors through Context Selection and Normalisation Tamara Polajnar University of Cambridge Computer Laboratory tp366@cam.ac.uk Stephen Clark University of Cambridge Computer Laboratory sc609@cam.ac.uk Abstract Distributional semantic models (DSMs) have been effective at representing seman- tics at the word level, and research has re- cently moved on to building distributional representations for larger segments of text. In this paper, we introduce novel ways of applying context selection and normalisa- tion to vary model sparsity and the range of values of the DSM vectors. We show how these methods enhance the quality of the vectors and thus result in improved low dimensional and composed represen- tations. We demonstrate these effects on standard word and phrase datasets, and on a new definition retrieval task and dataset. 1 Introduction Distributional semantic models (DSMs) (Turney and Pantel, 2010; Clarke, 2012) encode word meaning by counting co-occurrences with other words within a context window and recording these counts in a vector. Various IR and NLP tasks, such as word sense disambiguation, query expansion, and paraphrasing, take advantage of DSMs at a word level. More recently, researchers have been exploring methods that combine word vectors to represent phrases (Mitchell and Lapata, 2010; Baroni and Zamparelli, 2010) and sentences (Coecke et al., 2010; Socher et al., 2012). In this paper, we introduce two techniques that improve the quality of word vectors and can be easily tuned to adapt the vectors to particular lexical and com- positional tasks. The quality of the word vectors is generally as- sessed on standard datasets that consist of a list of word pairs and a corresponding list of gold stan- dard scores. These scores are gathered through an annotation task and reflect the similarity between the words as perceived by human judges (Bruni et al., 2012). Evaluation is conducted by comparing the word similarity predicted by the model with the gold standard using a correlation test such as Spearman’s ρ. While words, and perhaps some frequent shorter phrases, can be represented by distri- butional vectors learned through co-occurrence statistics, infrequent phrases and novel construc- tions are impossible to represent in that way. The goal of compositional DSMs is to find methods of combining word vectors, or perhaps higher-order tensors, into a single vector that represents the meaning of the whole segment of text. Elemen- tary approaches to composition employ simple op- erations, such as addition and elementwise prod- uct, directly on the word vectors. These have been shown to be effective for phrase similarity evalua- tion (Mitchell and Lapata, 2010) and detection of anomalous phrases (Kochmar and Briscoe, 2013). The methods that will be introduced in this pa- per can be applied to co-occurrence vectors to pro- duce improvements on word similarity and com- positional tasks with simple operators. We chose to examine the use of sum, elementwise prod- uct, and circular convolution (Jones …

Content truncated to 500 words