Evaluating Distributional Models of Semantics for Syntactically Invariant Inference

Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 33–43, Avignon, France, April 23 - 27 2012. c⃝2012 Association for Computational Linguistics Evaluating Distributional Models of Semantics for Syntactically Invariant Inference Jackie CK Cheung and Gerald Penn Department of Computer Science University of Toronto Toronto, ON, M5S 3G4, Canada {jcheung,gpenn}@cs.toronto.edu Abstract A major focus of current work in distri- butional models of semantics is to con- struct phrase representations composition- ally from word representations. However, the syntactic contexts which are modelled are usually severely limited, a fact which is reflected in the lexical-level WSD-like evaluation methods used. In this paper, we broaden the scope of these models to build sentence-level representations, and argue that phrase representations are best eval- uated in terms of the inference decisions that they support, invariant to the partic- ular syntactic constructions used to guide composition. We propose two evaluation methods in relation classification and QA which reflect these goals, and apply several recent compositional distributional models to the tasks. We find that the models out- perform a simple lemma overlap baseline slightly, demonstrating that distributional approaches can already be useful for tasks requiring deeper inference. 1 Introduction A number of unsupervised semantic models (Mitchell and Lapata, 2008, for example) have re- cently been proposed which are inspired at least in part by the distributional hypothesis (Harris, 1954)—that a word’s meaning can be character- ized by the contexts in which it appears. Such models represent word meaning as one or more high-dimensional vectors which capture the lex- ical and syntactic contexts of the word’s occur- rences in a training corpus. Much of the recent work in this area has, fol- lowing Mitchell and Lapata (2008), focused on the notion of compositionality as the litmus test of a truly semantic model. Compositionality is a nat- ural way to construct representations of linguistic units larger than a word, and it has a long history in Montagovian semantics for dealing with argu- ment structure and assembling rich semantical ex- pressions of the kind found in predicate logic. While compositionality may thus provide a convenient recipe for producing representations of propositionally typed phrases, it is not a nec- essary condition for a semantic representation. Rather, that distinction still belongs to the crucial ability to support inference. It is not the inten- tion of this paper to argue for or against composi- tionality in semantic representations. Rather, our interest is in evaluating semantic models in order to determine their suitability for inference tasks. In particular, we contend that it is desirable and arguably necessary for a compositional semantic representation to support inference invariantly, in the sense that the particular syntactic construction that guided the composition should not matter rel- ative to the representations of syntactically differ- ent phrases with the same meanings. For example, we can assert that John threw the ball and The ball was thrown by John have the same meaning for the purposes of inference, even though they differ syntactically. An …

Content truncated to 500 words