The Role of Syntax in Vector Space Models of Compositional Semantics

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 894–904, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics The Role of Syntax in Vector Space Models of Compositional Semantics Karl Moritz Hermann and Phil Blunsom Department of Computer Science University of Oxford Oxford, OX1 3QD, UK {karl.moritz.hermann,phil.blunsom}@cs.ox.ac.uk Abstract Modelling the compositional process by which the meaning of an utterance arises from the meaning of its parts is a funda- mental task of Natural Language Process- ing. In this paper we draw upon recent advances in the learning of vector space representations of sentential semantics and the transparent interface between syntax and semantics provided by Combinatory Categorial Grammar to introduce Com- binatory Categorial Autoencoders. This model leverages the CCG combinatory op- erators to guide a non-linear transforma- tion of meaning within a sentence. We use this model to learn high dimensional em- beddings for sentences and evaluate them in a range of tasks, demonstrating that the incorporation of syntax allows a con- cise model to learn representations that are both effective and general. 1 Introduction Since Frege stated his ‘Principle of Semantic Compositionality’ in 1892 researchers have pon- dered both how the meaning of a complex expres- sion is determined by the meanings of its parts, and how those parts are combined. (Frege, 1892; Pelletier, 1994). Over a hundred years on the choice of representational unit for this process of compositional semantics, and how these units combine, remain open questions. Frege’s principle may be debatable from a lin- guistic and philosophical standpoint, but it has provided a basis for a range of formal approaches to semantics which attempt to capture meaning in logical models. The Montague grammar (Mon- tague, 1970) is a prime example for this, build- ing a model of composition based on lambda- calculus and formal logic. More recent work in this field includes the Combinatory Categorial Grammar (CCG), which also places increased em- phasis on syntactic coverage (Szabolcsi, 1989). Recently those searching for the right represen- tation for compositional semantics have drawn in- spiration from the success of distributional mod- els of lexical semantics. This approach represents single words as distributional vectors, implying that a word’s meaning is a function of the envi- ronment it appears in, be that its syntactic role or co-occurrences with other words (Pereira et al., 1993; Sch¨utze, 1998). While distributional se- mantics is easily applied to single words, spar- sity implies that attempts to directly extract distri- butional representations for larger expressions are doomed to fail. Only in the past few years has it been attempted to extend these representations to semantic composition. Most approaches here use the idea of vector-matrix composition to learn larger representations from single-word encodings (Baroni and Zamparelli, 2010; Grefenstette and Sadrzadeh, 2011; Socher et al., 2012b). While these models have proved very promising for com- positional semantics, they make minimal use of linguistic information beyond the word level. In this paper we bridge the gap between recent advances in machine learning and …

Content truncated to 500 words