Easy Things First: Installments Improve Referring Expression Generation for Objects in Photographs

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 610–620, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Easy Things First: Installments Improve Referring Expression Generation for Objects in Photographs Sina Zarrieß David Schlangen Dialogue Systems Group // CITEC // Faculty of Linguistics and Literary Studies Bielefeld University, Germany first.last@uni-bielefeld.de Abstract Research on generating referring expres- sions has so far mostly focussed on “one- shot reference”, where the aim is to gener- ate a single, discriminating expression. In interactive settings, however, it is not un- common for reference to be established in “installments”, where referring informa- tion is offered piecewise until success has been confirmed. We show that this strat- egy can also be advantageous in technical systems that only have uncertain access to object attributes and categories. We train a recently introduced model of grounded word meaning on a data set of REs for objects in images and learn to predict se- mantically appropriate expressions. In a human evaluation, we observe that users are sensitive to inadequate object names - which unfortunately are not unlikely to be generated from low-level visual input. We propose a solution inspired from hu- man task-oriented interaction and imple- ment strategies for avoiding and repair- ing semantically inaccurate words. We enhance a word-based REG with context- aware, referential installments and find that they substantially improve the refer- ential success of the system. 1 Introduction A speaker who wants to refer to an object in a vi- sual scene will try to produce a referring expres- sion (RE) that (i) is semantically adequate, i.e. ac- curately describes the visual properties of the tar- get referent, and (ii) is pragmatically and contextu- ally appropriate, i.e. distinguishes the target from girl in front man on right anywhere brown Figure 1: Example images and REs from the ReferIt corpus (Kazemzadeh et al., 2014) other objects in the scene but does not overload the listener with unnecessary information. Figure 1 il- lustrates this with two examples from a corpus of REs collected from human subjects for objects in images (Kazemzadeh et al., 2014). Research on referring expression generation (REG) has mostly focussed on (ii), modeling prag- matic adequacy in attribute selection tasks, using as input a fully specified, symbolic representation of the visual attributes of an object and its distrac- tors in a scene (Dale and Reiter, 1995; Krahmer and Van Deemter, 2012). In this paper, we follow a more recent trend (Kazemzadeh et al., 2014; Gkatzia et al., 2015) and investigate REG on real-world images. In this setting, a low-level visual representation of an im- age (a scene) segmented into regions (objects), in- cluding the region of the target referent, consti- tutes the input. This task is closely related to the recently very active field of image-to-text gener- ation, where deep learning approaches have been used to directly map low-level visual input to nat- ural language sentences, e.g. (Vinyals et al., 2015; Chen and Lawrence Zitnick, 2015; Devlin et al., 2015). Similarly, …

Content truncated to 500 words