The REAL Corpus: A Crowd-Sourced Corpus of Human Generated and Evaluated Spatial References to Real-World Urban Scenes

The REAL Corpus: A Crowd-Sourced Corpus of Human Generated and Evaluated Spatial References to Real-World Urban Scenes Phil Bartie♢, William Mackaness♠, Dimitra Gkatzia⋆, Verena Rieser⋆ ♢School of Natural Sciences, ♠School of GeoSciences, ⋆Department of Computer Science University of Stirling, University of Edinburgh, Heriot-Watt University phil.bartie@stir.ac.uk, william.mackaness@ed.ac.uk, {d.gkatzia,v.t.rieser}@hw.ac.uk Abstract We present a newly crowd-sourced data set of natural language references to objects anchored in complex urban scenes (In short: The REAL Corpus – Referring Expressions Anchored Language). The REAL corpus contains a collection of images of real-world urban scenes together with verbal descriptions of target objects generated by humans, paired with data on how successful other people were able to identify the same object based on these descriptions. In total, the corpus contains 32 images with on average 27 descriptions per image and 3 verifications for each description. In addition, the corpus is annotated with a variety of linguistically motivated features. The paper highlights issues posed by collecting data using crowd-sourcing with an unrestricted input format, as well as using real-world urban scenes. The corpus will be released via the ELRA repository as part of this submission. Keywords: Image Descriptions, Spatial Referring Expressions, Urban Scenes, Vision and Language 1. Introduction Generating successful referring expressions (RE) is vital for real-world applications such as navigation systems. Tra- ditionally, research has focused on studying Referring Ex- pression Generation (REG) in virtual, controlled environ- ments. In this paper, we describe a novel corpus of spatial references from real scenes rather than virtual. Related work has focused on computer generated objects (van Deemter et al., 2006; Viethen and Dale, 2008), crafts (Mitchell et al., 2010), or small objects in a simple back- ground (Mitchell et al., 2013; Fitzgerald et al., 2013). One notable exception is the recent work by Kazemzadeh et al. (2014), who investigate referring expressions of objects in “complex photographs of real-world cluttered scenes”. They report that REs are heavily influenced by the object type. Here, we are interested in studying REs for visual objects in urban scenes. As the success of a RE is heav- ily dependent on the complexity of the scene as well as its linguistic features, we are interested in modelling and thus predicting the success of a RE. 2. REAL Corpus The REAL corpus contains a collection of images of real- world urban scenes (Fig. 1) together with verbal descrip- tions of target objects (see Fig. 2) generated by humans, paired with data on how successful other people were able to identify the same object based on these descriptions (Fig. 3). The data was collected through a web-based interface. The images were taken in Edinburgh (Scotland, UK) using a DSLR with a wide angle lens. The images were captured very early one summer morning to reduce the occlusion of city objects from buses and crowds, and to minimise light- ing and weather variations between images. 2.1. Experimental Setup There were 188 participants recruited (age between 16 to 71). Each participant was presented with an urban image (Fig. 1), where …

Content truncated to 500 words