Obtaining referential word meanings from visual and distributional information: Experiments on object naming

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 243–254 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1023 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 243–254 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1023 Obtaining referential word meanings from visual and distributional information: Experiments on object naming Sina Zarrieß and David Schlangen Dialogue Systems Group // CITEC // Faculty of Linguistics and Literary Studies Bielefeld University, Germany {sina.zarriess,david.schlangen}@uni-bielefeld.de Abstract We investigate object naming, which is an important sub-task of referring expression generation on real-world images. As op- posed to mutually exclusive labels used in object recognition, object names are more flexible, subject to communicative pref- erences and semantically related to each other. Therefore, we investigate models of referential word meaning that link vi- sual to lexical information which we as- sume to be given through distributional word embeddings. We present a model that learns individual predictors for object names that link visual and distributional aspects of word meaning during training. We show that this is particularly benefi- cial for zero-shot learning, as compared to projecting visual objects directly into the distributional space. In a standard object naming task, we find that different ways of combining lexical and visual information achieve very similar performance, though experiments on model combination sug- gest that they capture complementary as- pects of referential meaning. 1 Introduction Expressions referring to objects in visual scenes typically include a word naming the type of the object: E.g., house in Figure 1 (a), or, as a very general type, thingy in Figure 1 (d). Determin- ing such a name is a crucial step for referring expression generation (REG) systems, as many other decisions concerning, e.g., the selection of attributes follow from it (Dale and Reiter, 1995; Krahmer and Van Deemter, 2012). For a long time, however, research on REG mostly assumed the availability of symbolic representations of ref- (a)“house” (b)“buildings” (c)“large structure” (d)“roof thingy” Figure 1: Examples of object names in the REFERIT corpus referring to instances of buildings erent and scene, and sidestepped questions about how speakers actually choose these names, due to the lack of models capable of capturing what a word like house refers to in the real world. Recent advances in image processing promise to fill this gap, with state-of-the-art computer vi- sion systems being able to classify images into thousands of different categories (e.g. Szegedy et al. (2015)). However, classification is not nam- ing (Ordonez et al., 2016). Standard object clas- sification schemes are inherently “flat”, and treat object labels as mutually exclusive (Deng et al., 2014). A state-of-the-art object recognition sys- tem would be trained to classify the object in e.g. Figure 1 (a) as either house or building, ignoring the lexical similarity between these two names. In contrast, humans seem to be more flexible as to the chosen level of generality. Depending on the prototypicality of the object to name, …

Content truncated to 500 words