Combining Geometric, Textual and Visual Features for Predicting Prepositions in Image Descriptions

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 214–220, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Combining Geometric, Textual and Visual Features for Predicting Prepositions in Image Descriptions Arnau Ramisa*1 Josiah Wang*2 Ying Lu3 Emmanuel Dellandrea3 Francesc Moreno-Noguer1 Robert Gaizauskas2 1 Institut de Rob`otica i Inform`atica Industrial (UPC-CSIC), Barcelona, Spain 2 Department of Computer Science, University of Sheffield, UK 3 LIRIS, ´Ecole Centrale de Lyon, France {aramisa, fmoreno}@iri.upc.edu {j.k.wang, r.gaizauskas}@sheffield.ac.uk {ying.lu, emmanuel.dellandrea}@ec-lyon.fr Abstract We investigate the role that geometric, tex- tual and visual features play in the task of predicting a preposition that links two visual entities depicted in an image. The task is an important part of the subsequent process of generating image descriptions. We explore the prediction of prepositions for a pair of entities, both in the case when the labels of such entities are known and unknown. In all situations we found clear evidence that all three features contribute to the prediction task. 1 Introduction In recent years, there has been an increased in- terest in the task of automatic generation of natu- ral language image descriptions at sentence level, compared to earlier work that annotates images with a laundry list of terms (Duygulu et al., 2002). The task is important in that such detailed anno- tations are more informative and discriminative compared to isolated textual labels, and are essen- tial for improved text and image retrieval. The most standard approach to generating such descriptions involves first detecting instances of pre-defined concepts in the image, and then rea- soning about these concepts to generate image de- scriptions e.g. (Kulkarni et al., 2011; Yang et al., 2011). Our work is also based on this paradigm. However, we assume that object instances have already been pre-detected by visual recognisers, and concentrate on a specific subtask of descrip- tion generation. More specifically, given two vi- sual entity instances where one could potentially act as a modifier to the other, we address the prob- lem of identifying the appropriate preposition to connect these two entities (Figure 1). The inferred prepositional relations will subsequently act as an *A. Ramisa and J. Wang contributed equally to this work. Figure 1: Given a subject boy and an object sled and their location in the image, what would the best preposition be to connect the two entities? important intermediate representation towards the eventual goal of generating image descriptions. The main contribution of this paper is therefore to learn to predict the most suitable preposition given its context, and to learn this jointly from im- ages and their descriptions. In particular, we con- centrate on learning from (i) geometric relations between two visual entities from image annota- tions; (ii) textual features from textual descrip- tions; (iii) visual features from images. Previous work exists (Yang et al., 2011) that uses text cor- pora to ‘guess’ the prepositions given the context without considering the appropriate spatial rela- tions between the entities in the image, signifying a gap between visual content …

Content truncated to 500 words