Modelling Word Meaning using Efficient Tensor Representations

Modelling Word Meaning using Efficient Tensor Representations Mike Symonds a, Peter Bruza a, Laurianne Sitbon a, and Ian Turner b aDepartment of Computer Science, Queensland University of Technology, Brisbane, Queensland, Australia {m.symonds, p.bruza, l.sitbon}@qut.edu.au bDepartment of Mathematics, Queensland University of Technology, Brisbane, Queensland, Australia i.turner@qut.edu.au Abstract. Models of word meaning, built from a corpus of text, have demonstrated success in emulating human performance on a number of cognitive tasks. Many of these models use geometric representations of words to store semantic associations between words. Often word order information is not captured in these models. The lack of structural information used by these models has been raised as a weakness when performing cognitive tasks. This paper presents an efficient tensor based approach to modelling word meaning that builds on recent attempts to encode word order information, while providing flexible methods for extracting task specific semantic information. Keywords: Semantic Space, Unsupervised Learning, Tensor, Word Order 1 Introduction Research in the area of natural language processing has demonstrated that psychologically relevant models of word meaning can be learnt from exposure to natural language (Lan- dauer and Dumais, 1997; Lund and Burgess, 1996; McRoy, 1992; Turney, 2008). Many of these models are based on vector representations built from word co-occurrence statis- tics that aim to model various semantic relationships. Even though these semantic space models appear to identify words with similar meanings, it has been argued that they do not incorporate syntax or achieve other basic cognitive language abilities (Perfetti, 1998). Recently, a number of semantic space models, that learn directly from unstructured text, have been developed that encode word order into the semantic space, hence capturing more structural information about word associations (Jones and Mewhort, 2007; Sahlgren et al., 2008). Jones and Mewhort (2007) concluded that a model that pays attention to both context and word order while learning, stands a greater chance of matching the trends found in human data. The strength of a geometric approach to encode word order is in the ability to work within a mathematically well defined framework, including the availability of many existing operators from linear algebra, such as Kronecker products. However, to our knowledge there has been very few efficient methods for implementing uncompressed Kronecker products when encoding word order information within a semantic space. The main contribution of this paper is to present a novel, efficient approach to using Kronecker products to encode word order information within a semantic space. The other significant contribution is to demonstrate how applications can use our single representa- tion to access various task specific semantic information. Copyright 2011 by Mike Symonds, Peter Bruza, Laurianne Sitbon, and Ian Turner 25th Pacific Asia Conference on Language, Information and Computation, pages 313–322 313 2 Related Work The main areas of research that provide a theoretical framework for our model include: (i) the structuralist approaches to defining word meaning, and (ii) the use of semantic spaces to model word meaning. 2.1 Word Meaning Ferdinand de Saussure (1916) argued that meaning arose from the …

Content truncated to 500 words