From the Virtual to the RealWorld: Referring to Objects in Real-World Spatial Scenes

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1936–1942, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. From the Virtual to the Real World: Referring to Objects in Real-World Spatial Scenes Dimitra Gkatzia, Verena Rieser Department of Computer Science Heriot-Watt University Edinburgh EH14 4AS , UK {d.gkatzia,v.t.rieser}@hw.ac.uk Phil Bartie School of Natural Sciences University of Stirling Stirling FK9 4LA, UK phil.bartie@stir.ac.uk William Mackaness School of GeoSciences University of Edinburgh Edinburgh EH8 9XP, UK william.mackaness@ed.ac.uk Abstract Predicting the success of referring expres- sions (RE) is vital for real-world applica- tions such as navigation systems. Tradi- tionally, research has focused on studying Referring Expression Generation (REG) in virtual, controlled environments. In this paper, we describe a novel study of spa- tial references from real scenes rather than virtual. First, we investigate how humans describe objects in open, uncontrolled sce- narios and compare our findings to those reported in virtual environments. We show that REs in real-world scenarios differ sig- nificantly to those in virtual worlds. Sec- ond, we propose a novel approach to quan- tifying image complexity when complete annotations are not present (e.g. due to poor object recognition capabitlities), and third, we present a model for success pre- diction of REs for objects in real scenes. Finally, we discuss implications for Nat- ural Language Generation (NLG) systems and future directions. 1 Introduction REG has attracted considerable interest in the NLG community over the past 20 years (Krahmer and van Deemter, 2011; Gatt et al., 2014). While initially, the standard evaluation metric for REG was human-likeness, as compared to human cor- pora similarity as in TUNA (Gatt et al., 2009), the field has moved on to evaluating REG effec- tiveness by measuring task success in virtual in- teractive environments (Byron et al., 2009; Gar- gett et al., 2010; Janarthanam et al., 2012). Vir- tual environments however eliminate real-world uncertainty, such object recognition errors or clut- tered scenes. In this paper, we investigate whether the lessons learnt in virtual environments can be transferred to real-world scenes. We consider the case where we are uncertain about the scene itself, i.e. we assume that the complexity of the scene is hidden and we are interested in identifying a specific object, and thus our work differs from approaches that generate descriptions for images such as (Mitchell et al., 2012; Feng and Lapata, 2013; Yang et al., 2011; Yatskar et al., 2014). Related work has focused on computer gener- ated objects (van Deemter et al., 2006; Viethen and Dale, 2008), crafts (Mitchell et al., 2010), or small objects in a simple background (Mitchell et al., 2013a; FitzGerald et al., 2013). One notable exception is the recent work by Kazemzadeh et al. (2014), who investigate referring expressions of objects in “complex photographs of real-world cluttered scenes”. They report that REs are heavily influenced by the object type. Here, we are inter- ested in studying REs for visual objects in urban scenes. As the success of a RE is heavily depen- dent …

Content truncated to 500 words