Prior Disambiguation of Word Tensors for Constructing Sentence Vectors

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1590–1601, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Prior Disambiguation of Word Tensors for Constructing Sentence Vectors Dimitri Kartsaklis University of Oxford Department of Computer Science Wolfson Building, Parks Road Oxford, OX1 3QD, UK dimitri.kartsaklis@cs.ox.ac.uk Mehrnoosh Sadrzadeh Queen Mary University of London School of Electronic Engineering and Computer Science Mile End Road London, E1 4NS, UK mehrs@eecs.qmul.ac.uk Abstract Recent work has shown that compositional- distributional models using element-wise op- erations on contextual word vectors benefit from the introduction of a prior disambigua- tion step. The purpose of this paper is to generalise these ideas to tensor-based models, where relational words such as verbs and ad- jectives are represented by linear maps (higher order tensors) acting on a number of argu- ments (vectors). We propose disambiguation algorithms for a number of tensor-based mod- els, which we then test on a variety of tasks. The results show that disambiguation can pro- vide better compositional representation even for the case of tensor-based models. Further- more, we confirm previous findings regarding the positive effect of disambiguation on vec- tor mixture models, and we compare the ef- fectiveness of the two approaches. 1 Introduction Distributional models of meaning have been proved extremely useful for a number of natural language processing tasks, ranging from thesaurus extraction (Curran, 2004) to topic modelling (Landauer and Dumais, 1997) and information retrieval (Manning et al., 2008), to name just a few. These models are based on the distributional hypothesis of Har- ris (1968), which states that the meaning of a word depends on its context. This idea allows the words to be represented by vectors of statistics collected from a sufficiently large corpus of text; each ele- ment of the vector reflects how many times a word co-occurs in the same context with another word of the vocabulary. However, due to the generative power of natural language, which is able to pro- duce infinite new structures from a finite set of re- sources (words), no text corpus, regardless of its size, can provide reliable distributional representa- tions for anything longer than single words or per- haps very short phrases consisting of two words; in other words, this technique cannot scale up to the phrase or sentence level. Much research activity has been recently dedi- cated to provide a solution to this problem: although the direct construction of a sentence vector is not possible, we might still be able to synthetically cre- ate such a vectorial representation by somehow com- posing the vectors of the words that comprise the sentence. Towards this goal, researchers have em- ployed a variety of approaches that roughly fall into two general categories. Following an influential work (Mitchell and Lapata, 2008), the models in the first category compute a sentence vector as a mix- ture of the original word vectors, using simple oper- ations such as element-wise multiplication and ad- dition; we refer to these models as vector mixtures. …

Content truncated to 500 words