Vector-based Models of Semantic Composition

Proceedings of ACL-08: HLT, pages 236–244, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Vector-based Models of Semantic Composition Jeff Mitchell and Mirella Lapata School of Informatics, University of Edinburgh 2 Buccleuch Place, Edinburgh EH8 9LW, UK jeff.mitchell@ed.ac.uk, mlap@inf.ed.ac.uk Abstract This paper proposes a framework for repre- senting the meaning of phrases and sentences in vector space. Central to our approach is vector composition which we operationalize in terms of additive and multiplicative func- tions. Under this framework, we introduce a wide range of composition models which we evaluate empirically on a sentence similarity task. Experimental results demonstrate that the multiplicative models are superior to the additive alternatives when compared against human judgments. 1 Introduction Vector-based models of word meaning (Lund and Burgess, 1996; Landauer and Dumais, 1997) have become increasingly popular in natural language processing (NLP) and cognitive science. The ap- peal of these models lies in their ability to rep- resent meaning simply by using distributional in- formation under the assumption that words occur- ring within similar contexts are semantically similar (Harris, 1968). A variety of NLP tasks have made good use of vector-based models. Examples include au- tomatic thesaurus extraction (Grefenstette, 1994), word sense discrimination (Sch¨utze, 1998) and dis- ambiguation (McCarthy et al., 2004), collocation ex- traction (Schone and Jurafsky, 2001), text segmen- tation (Choi et al., 2001) , and notably information retrieval (Salton et al., 1975). In cognitive science vector-based models have been successful in simu- lating semantic priming (Lund and Burgess, 1996; Landauer and Dumais, 1997) and text comprehen- sion (Landauer and Dumais, 1997; Foltz et al., 1998). Moreover, the vector similarities within such semantic spaces have been shown to substantially correlate with human similarity judgments (McDon- ald, 2000) and word association norms (Denhire and Lemaire, 2004). Despite their widespread use, vector-based mod- els are typically directed at representing words in isolation and methods for constructing representa- tions for phrases or sentences have received little attention in the literature. In fact, the common- est method for combining the vectors is to average them. Vector averaging is unfortunately insensitive to word order, and more generally syntactic struc- ture, giving the same representation to any construc- tions that happen to share the same vocabulary. This is illustrated in the example below taken from Lan- dauer et al. (1997). Sentences (1-a) and (1-b) con- tain exactly the same set of words but their meaning is entirely different. (1) a. It was not the sales manager who hit the bottle that day, but the office worker with the serious drinking problem. b. That day the office manager, who was drinking, hit the problem sales worker with a bottle, but it was not serious. While vector addition has been effective in some applications such as essay grading (Landauer and Dumais, 1997) and coherence assessment (Foltz et al., 1998), there is ample empirical evidence that syntactic relations across and within sentences are crucial for sentence and discourse processing (Neville et al., 1991; West and Stanovich, 1986) and modulate cognitive …

Content truncated to 500 words