Refer360∘: A Referring Expression Recognition Dataset in 360∘ Images

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7189–7202 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7189 Refer360° : A Referring Expression Recognition Dataset in 360° Images Volkan Cirik1 Taylor Berg-Kirkpatrick2 Louis-Philippe Morency1 1Carnegie Mellon University 2University of California San Diego {vcirik,morency}@cs.cmu.edu {tberg}@eng.ucsd.edu Abstract We propose a novel large-scale referring ex- pression recognition dataset, Refer360°, con- sisting of 17,137 instruction sequences and ground-truth actions for completing these in- structions in 360° scenes. Refer360° differs from existing related datasets in three ways. First, we propose a more realistic scenario where instructors and the followers have par- tial, yet dynamic, views of the scene – fol- lowers continuously modify their field-of-view (FoV) while interpreting instructions that spec- ify a final target location. Second, instructions to find the target location consist of multiple steps for followers who will start at random FoVs. As a result, intermediate instructions are strongly grounded in object references and followers must identify intermediate FoVs to find the final target location correctly. Third, the target locations are neither restricted to predefined objects nor chosen by annotators; instead, they are distributed randomly across scenes. This “point anywhere” approach leads to more linguistically complex instructions, as shown in our analyses. Our examination of the dataset shows that Refer360° manifests linguistically rich phenomena in a language grounding task that poses novel challenges for computational modeling of language, vision, and navigation. 1 Introduction Imagine a scenario in which you are asked to re- trieve medication from a bathroom. ‘First, face the sink, then find the second drawer in the cab- inet to your left. The pills should be inside that drawer behind the toothbrush.” Interpreting in- struction sequences in order to locate targets in novel environments is challenging for AI systems (e.g. personal robots and self-driving cars). First, the system needs to ground the instructions into visual perception (Anderson et al., 2018b; Hu et al., look$towards$the$ door$leading$ outside$the$cafe. notice the silver and black coffee pot closest to you on the bar. see the black trash bin on the floor in front of the coffee pot. waldo is on the face of the trash bin about 1 foot off the floor and also slightly on the brown wood. 0 1 2 3 Random$Starting$Point Figure 1: An example from Refer360° . Orange frames represent the field-of-view (FoV) of the follower after interpreting each instruction. Numbers in the frames represent the sequential order. Green lines show how FoVs change continuously. After each instruction, the follower changes the FoV to align with what the instruc- tion describes. Please see Figure 2a to see the correct location of Waldo. 2019). This often requires identification of the mentioned object (Plummer et al., 2015) through physical relationships with surrounding objects (Hu et al., 2017b; Cirik et al., 2018a). Second, since human visual perception has limited field-of-view, instructions are often sequential: First, the correct FoV should be identified before searching for the final target. In many situations, the target loca- tion is not …

Content truncated to 500 words