Compositional Matrix-Space Models of Language

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 907–916, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics Compositional Matrix-Space Models of Language Sebastian Rudolph Karlsruhe Institute of Technology Karlsruhe, Germany rudolph@kit.edu Eugenie Giesbrecht FZI Forschungszentrum Informatik Karlsuhe, Germany giesbrecht@fzi.de Abstract We propose CMSMs, a novel type of generic compositional models for syntac- tic and semantic aspects of natural lan- guage, based on matrix multiplication. We argue for the structural and cognitive plau- sibility of this model and show that it is able to cover and combine various com- mon compositional NLP approaches rang- ing from statistical word space models to symbolic grammar formalisms. 1 Introduction In computational linguistics and information re- trieval, Vector Space Models (Salton et al., 1975) and its variations – such as Word Space Models (Schütze, 1993), Hyperspace Analogue to Lan- guage (Lund and Burgess, 1996), or Latent Se- mantic Analysis (Deerwester et al., 1990) – have become a mainstream paradigm for text represen- tation. Vector Space Models (VSMs) have been empirically justified by results from cognitive sci- ence (Gärdenfors, 2000). They embody the distri- butional hypothesis of meaning (Firth, 1957), ac- cording to which the meaning of words is defined by contexts in which they (co-)occur. Depending on the specific model employed, these contexts can be either local (the co-occurring words), or global (a sentence or a paragraph or the whole doc- ument). Indeed, VSMs proved to perform well in a number of tasks requiring computation of seman- tic relatedness between words, such as synonymy identification (Landauer and Dumais, 1997), auto- matic thesaurus construction (Grefenstette, 1994), semantic priming, and word sense disambiguation (Padó and Lapata, 2007). Until recently, little attention has been paid to the task of modeling more complex concep- tual structures with such models, which consti- tutes a crucial barrier for semantic vector models on the way to model language (Widdows, 2008). An emerging area of research receiving more and more attention among the advocates of distribu- tional models addresses the methods, algorithms, and evaluation strategies for representing compo- sitional aspects of language within a VSM frame- work. This requires novel modeling paradigms, as most VSMs have been predominantly used for meaning representation of single words and the key problem of common bag-of-words-based VSMs is that word order information and thereby the structure of the language is lost. There are approaches under way to work out a combined framework for meaning representa- tion using both the advantages of symbolic and distributional methods. Clark and Pulman (2007) suggest a conceptual model which unites sym- bolic and distributional representations by means of traversing the parse tree of a sentence and ap- plying a tensor product for combining vectors of the meanings of words with the vectors of their roles. The model is further elaborated by Clark et al. (2008). To overcome the aforementioned difficulties with VSMs and work towards a tight integra- tion of symbolic and distributional approaches, we propose a Compositional Matrix-Space Model (CMSM) which employs matrices instead …

Content truncated to 500 words