Semantic Compositionality through Recursive Matrix-Vector Spaces

Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1201–1211, Jeju Island, Korea, 12–14 July 2012. c⃝2012 Association for Computational Linguistics Semantic Compositionality through Recursive Matrix-Vector Spaces Richard Socher Brody Huval Christopher D. Manning Andrew Y. Ng richard@socher.org, {brodyh,manning,ang}@stanford.edu Computer Science Department, Stanford University Abstract Single-word vector space models have been very successful at learning lexical informa- tion. However, they cannot capture the com- positional meaning of longer phrases, prevent- ing them from a deeper understanding of lan- guage. We introduce a recursive neural net- work (RNN) model that learns compositional vector representations for phrases and sen- tences of arbitrary syntactic type and length. Our model assigns a vector and a matrix to ev- ery node in a parse tree: the vector captures the inherent meaning of the constituent, while the matrix captures how it changes the mean- ing of neighboring words or phrases. This matrix-vector RNN can learn the meaning of operators in propositional logic and natural language. The model obtains state of the art performance on three different experiments: predicting fine-grained sentiment distributions of adverb-adjective pairs; classifying senti- ment labels of movie reviews and classifying semantic relationships such as cause-effect or topic-message between nouns using the syn- tactic path between them. 1 Introduction Semantic word vector spaces are at the core of many useful natural language applications such as search query expansions (Jones et al., 2006), fact extrac- tion for information retrieval (Pas¸ca et al., 2006) and automatic annotation of text with disambiguated Wikipedia links (Ratinov et al., 2011), among many others (Turney and Pantel, 2010). In these mod- els the meaning of a word is encoded as a vector computed from co-occurrence statistics of a word and its neighboring words. Such vectors have been shown to correlate well with human judgments of word similarity (Griffiths et al., 2007). … very good movie ... ( a , A ) ( b , B ) ( c , C ) Recursive Matrix-Vector Model f(Ba, Ab)= Ba= Ab= - vector - matrix ... … Figure 1: A recursive neural network which learns se- mantic vector representations of phrases in a tree struc- ture. Each word and phrase is represented by a vector and a matrix, e.g., very = (a, A). The matrix is applied to neighboring vectors. The same function is repeated to combine the phrase very good with movie. Despite their success, single word vector models are severely limited since they do not capture com- positionality, the important quality of natural lan- guage that allows speakers to determine the meaning of a longer expression based on the meanings of its words and the rules used to combine them (Frege, 1892). This prevents them from gaining a deeper understanding of the semantics of longer phrases or sentences. Recently, there has been much progress in capturing compositionality in vector spaces, e.g., (Mitchell and Lapata, 2010; Baroni and Zamparelli, 2010; Zanzotto et al., 2010; Yessenalina and Cardie, 2011; Socher et al., 2011c) …

Content truncated to 500 words