Resolving References to Objects in Photographs using the Words-As-Classifiers Model

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1213–1223, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Resolving References to Objects in Photographs using the Words-As-Classifiers Model David Schlangen Sina Zarrieß Casey Kennington Dialogue Systems Group // CITEC // Faculty of Linguistics and Literary Studies Bielefeld University, Germany first.last@uni-bielefeld.de Abstract A common use of language is to refer to visually present objects. Modelling it in computers requires modelling the link between language and perception. The “words as classifiers” model of grounded semantics views words as classifiers of perceptual contexts, and composes the meaning of a phrase through composition of the denotations of its component words. It was recently shown to perform well in a game-playing scenario with a small num- ber of object types. We apply it to two large sets of real-world photographs that contain a much larger variety of object types and for which referring expressions are available. Using a pre-trained convolu- tional neural network to extract image re- gion features, and augmenting these with positional information, we show that the model achieves performance competitive with the state of the art in a reference res- olution task (given expression, find bound- ing box of its referent), while, as we argue, being conceptually simpler and more flex- ible. 1 Introduction A common use of language is to refer to objects in the shared environment of speaker and addressee. Being able to simulate this is of particular impor- tance for verbal human/robot interfaces (HRI), and the task has consequently received some attention in this field (Matuszek et al., 2012; Tellex et al., 2011; Krishnamurthy and Kollar, 2013). Here, we study a somewhat simpler precursor task, namely that of resolution of reference to objects in static images (photographs), but use a larger set of object types than is usually done in HRI work (> 300, see below). More formally, the task is to retrieve, given a referring expression e and an image I, the region bb∗ of the image that is most likely to contain the referent of the expres- sion. As candidate regions, we use both manually annotated regions as well as automatically com- puted ones. As our starting point, we use the “words-as- classifiers” model recently proposed by Kenning- ton and Schlangen (2015). It has before only been tested in a small domain and with specially de- signed features; here, we apply it to real-world photographs and use learned representations from a convolutional neural network (Szegedy et al., 2015). We learn models for between 400 and 1,200 words, depending on the training data set. As we show, the model performs competitive with the state of the art (Hu et al., 2016; Mao et al., 2016) on the same data sets. Our background interest in situated interaction makes it important for us that the approach we use is ‘dialogue ready’; and it is, in the sense that it supports incremental processing (giving re- sults while the incoming utterance is going on) and incremental …

Content truncated to 500 words