ReferItGame: Referring to Objects in Photographs of Natural Scenes

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 787–798, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics ReferItGame: Referring to Objects in Photographs of Natural Scenes Sahar Kazemzadeh1∗ Vicente Ordonez1∗ Mark Matten2 Tamara L. Berg1 1University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, USA 2The Bishop’s School, San Diego, CA 92037, USA vicente@cs.unc.edu,tlberg@cs.unc.edu Abstract In this paper we introduce a new game to crowd-source natural language referring expressions. By designing a two player game, we can both collect and verify refer- ring expressions directly within the game. To date, the game has produced a dataset containing 130,525 expressions, referring to 96,654 distinct objects, in 19,894 pho- tographs of natural scenes. This dataset is larger and more varied than previous REG datasets and allows us to study referring expressions in real-world scenes. We pro- vide an in depth analysis of the resulting dataset. Based on our findings, we design a new optimization based model for gen- erating referring expressions and perform experimental evaluations on 3 test sets. 1 Introduction Much of everyday language and discourse con- cerns the visual world around us, making under- standing the relationship between objects in the physical world and language describing those ob- jects an important challenge problem for AI. From robotics, to image search, to situated language learning, and natural language grounding, there are a number of research areas that would bene- fit from a better understanding of how people refer to physical entities in the world. Recent advances in automatic computer vision methods have started to make technologies for rec- ognizing thousands of object categories a near re- ality (Perronnin et al., 2012; Deng et al., 2012; Deng et al., 2010; Krizhevsky et al., 2012). As a result, there has been a spurt of recent work trying to estimate higher level semantics, including ex- citing efforts to automatically produce natural lan- guage descriptions of images and video (Farhadi et ∗Indicates equal author contribution. al., 2010; Kulkarni et al., 2011; Yang et al., 2011; Ordonez et al., 2011; Kuznetsova et al., 2012; Feng and Lapata, 2013). Common challenges en- countered in these pursuits include the fact that descriptions can be highly task dependent, open- ended, and difficult to evaluate automatically. Therefore, we look at the related, but more fo- cused problem of referring expression generation (REG). Previous work on REG has made signif- icant progress toward understanding how people generate expressions to refer to objects (a recent survey of techniques is provided in Krahmer and van Deemter (2012)). In this paper, we study the relatively unexplored setting of how people refer to objects in complex photographs of real-world cluttered scenes. One initial stumbling block to examining this scenario is lack of existing rele- vant datasets, as previous collections for studying REG have used relatively focused domains such as graphics generated objects (van Deemter et al., 2006; Viethen and Dale, 2008), crafts (Mitchell et al., 2010), or small everyday (home and office) ob- jects arrayed …

Content truncated to 500 words