Information Structure Prediction for Visual-world Referring Expressions

Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 520–529, Gothenburg, Sweden, April 26-30 2014. c⃝2014 Association for Computational Linguistics Information Structure Prediction for Visual-world Referring Expressions Micha Elsner Department of Linguistics, The Ohio State University melsner@ling.osu.edu Hannah Rohde Linguistics & English Language, University of Edinburgh hannah.rohde@ed.ac.uk Alasdair D. F. Clarke School of Informatics, University of Edinburgh a.clarke@ed.ac.uk Abstract We investigate the order of mention for objects in relational descriptions in visual scenes. Existing work in the visual do- main focuses on content selection for text generation and relies primarily on tem- plates to generate surface realizations from underlying content choices. In contrast, we seek to clarify the influence of visual perception on the linguistic form (as op- posed to the content) of descriptions, mod- eling the variation in and constraints on the surface orderings in a description. We find previously-unknown effects of the vi- sual characteristics of objects; specifically, when a relational description involves a vi- sually salient object, that object is more likely to be mentioned first. We conduct a detailed analysis of these patterns using logistic regression, and also train and eval- uate a classifier. Our methods yield signif- icant improvement in classification accu- racy over a naive baseline. 1 Introduction Visual-world referring expression generation (REG) is the task of instructing a listener how to find an object (the target) in a visual scene. In complicated scenes, people often produce relational descriptions, in which the target object is described relative to another (a landmark) (Viethen and Dale, 2008). While existing REG systems can generate relational descriptions, they tend to focus on content selection (that is, choosing an appropriate set of landmarks for each object). Surface realization (turning the selected content into a string of words) is handled by simple heuristics, such as sets of templates. Complex descriptions, however, have a non-trivial information structure— objects are not mentioned in an arbitrary order. Numerous studies in non-visual domains show that English speakers favor constructions that place familiar (given) information before unfamiliar (new) (Bresnan et al., 2007; Ward and Birner, 2001; Prince, 1981). We show that this pattern also holds for visual- world referring expressions (REs), and moreover, that objects with sufficient visual prominence are treated as given. Thus, we argue that the concept of salience used in surface realization should incorporate metrics from visual perception. In this study, we create a model of information ordering in complex relational descriptions. Us- ing a discriminative classifier, we learn to predict the information structuring strategies used in our corpus. We compare these strategies to the typical given/new pattern of English discourse. Experi- ments on a corpus of descriptions of cartoon peo- ple in the childrens’ book “Where’s Wally” (Hand- ford, 1987), corpus described in (Clarke et al., 2013), show that our approach significantly out- performs a naive baseline, improving especially on prediction of non-canonical orderings. This study has three main contributions. First, it demonstrates that humans use sophisticated in- formation ordering strategies for REG, and …

Content truncated to 500 words