A Comparison of Vector-based Representations for Semantic Composition

Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 546–556, Jeju Island, Korea, 12–14 July 2012. c⃝2012 Association for Computational Linguistics A Comparison of Vector-based Representations for Semantic Composition William Blacoe and Mirella Lapata Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB w.b.blacoe@sms.ed.ac.uk, mlap@inf.ed.ac.uk Abstract In this paper we address the problem of modeling compositional meaning for phrases and sentences using distributional methods. We experiment with several possible com- binations of representation and composition, exhibiting varying degrees of sophistication. Some are shallow while others operate over syntactic structure, rely on parameter learn- ing, or require access to very large corpora. We find that shallow approaches are as good as more computationally intensive alternatives with regards to two particular tests: (1) phrase similarity and (2) paraphrase detection. The sizes of the involved training corpora and the generated vectors are not as important as the fit between the meaning representation and compositional method. 1 Introduction Distributional models of semantics have seen con- siderable success at simulating a wide range of be- havioral data in tasks involving semantic cognition and also in practical applications. For example, they have been used to model judgments of semantic sim- ilarity (McDonald, 2000) and association (Denhire and Lemaire, 2004; Griffiths et al., 2007) and have been shown to achieve human level performance on synonymy tests (Landauer and Dumais, 1997; Griffiths et al., 2007) such as those included in the Test of English as a Foreign Language (TOEFL). This ability has been put to practical use in numer- ous natural language processing tasks such as au- tomatic thesaurus extraction (Grefenstette, 1994), word sense discrimination (Sch¨utze, 1998), lan- guage modeling (Bellegarda, 2000), and the iden- tification of analogical relations (Turney, 2006). While much research has been directed at the most effective ways of constructing representations for individual words, there has been far less con- sensus regarding the representation of larger con- structions such as phrases and sentences. The prob- lem has received some attention in the connection- ist literature, particularly in response to criticisms of the ability of connectionist representations to handle complex structures (Smolensky, 1990; Plate, 1995). More recently, several proposals have been put for- ward for computing the meaning of word combina- tions in vector spaces. This renewed interest is partly due to the popularity of distributional methods and their application potential to tasks that require an un- derstanding of larger phrases or complete sentences. For example, Mitchell and Lapata (2010) intro- duce a general framework for studying vector com- position, which they formulate as a function f of two vectors u and v. Different composition mod- els arise, depending on how f is chosen. Assuming that composition is a linear function of the Cartesian product of u and v allows to specify additive mod- els which are by far the most common method of vector combination in the literature (Landauer and Dumais, 1997; Foltz et al., …

Content truncated to 500 words