Simple Learning and Compositional Application of Perceptually Grounded Word Meanings for Incremental Reference Resolution

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 292–301, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Simple Learning and Compositional Application of Perceptually Grounded Word Meanings for Incremental Reference Resolution Casey Kennington CITEC, Bielefeld University Universit¨atsstraße 25 33615 Bielefeld, Germany ckennington@cit-ec. uni-bielefeld.de David Schlangen CITEC, Bielefeld University Universit¨atsstraße 25 33615 Bielefeld, Germany david.schlangen@ uni-bielefeld.de Abstract An elementary way of using language is to refer to objects. Often, these objects are physically present in the shared envi- ronment and reference is done via men- tion of perceivable properties of the ob- jects. This is a type of language use that is modelled well neither by logical semantics nor by distributional semantics, the former focusing on inferential relations between expressed propositions, the latter on simi- larity relations between words or phrases. We present an account of word and phrase meaning that is perceptually grounded, trainable, compositional, and ‘dialogue- plausible’ in that it computes meanings word-by-word. We show that the approach performs well (with an accuracy of 65% on a 1-out-of-32 reference resolution task) on direct descriptions and target/landmark descriptions, even when trained with less than 800 training examples and automati- cally transcribed utterances. 1 Introduction The most basic, fundamental site of language use is co-located dialogue (Fillmore, 1975; Clark, 1996) and referring to objects, as in Example (1), is a common occurrence in such a co-located set- ting. (1) The green book on the left next to the mug. Logical semantics (Montague, 1973; Gamut, 1991; Partee et al., 1993) has little to say about this process – its focus is on the construction of syntactically manipulable objects that model infer- ential relations; here, e.g. the inference that there are (at least) two objects. Vector space approaches to distributional semantics (Turney and Pantel, 2010) similarly focuses on something else, namely semantic similarity relations between words or phrases (e.g. finding closeness for “coloured tome on the right of the cup”). Neither approach by it- self says anything about processing; typically, the assumption in applications is that fully presented phrases are being processed. Lacking in these approaches is a notion of grounding of symbols in features of the world (Harnad, 1990).1 In this paper, we present an ac- count of word and phrase meaning that is (a) per- ceptually grounded in that it provides a link be- tween words and (computer) vision features of real images, (b) trainable, as that link is learned from examples of language use, (c) compositional in that the meaning of phrases is a function of that of its parts and composition is driven by structural analysis, and (d) ‘dialogue-plausible’ in that it computes meanings incrementally, word-by-word and can work with noisy input from an automatic speech recogniser (ASR). We show that the ap- proach performs well (with an accuracy of 65% on a reference resolution task out of 32 objects) on direct descriptions as well as target/landmark de- scriptions, even when trained …

Content truncated to 500 words