Where is Misty? Interpreting Spatial Descriptors by Modeling Regions in Space

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 157–166 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Where is Misty? Interpreting Spatial Descriptors by Modeling Regions in Space Nikita Kitaev and Dan Klein Computer Science Division University of California, Berkeley {kitaev,klein}@cs.berkeley.edu Abstract We present a model for locating regions in space based on natural language de- scriptions. Starting with a 3D scene and a sentence, our model is able to associate words in the sentence with regions in the scene, interpret relations such as on top of or next to, and finally locate the re- gion described in the sentence. All com- ponents form a single neural network that is trained end-to-end without prior knowl- edge of object segmentation. To evalu- ate our model, we construct and release a new dataset consisting of Minecraft scenes with crowdsourced natural language de- scriptions. We achieve a 32% relative er- ror reduction compared to a strong neural baseline. 1 Introduction In this work, we present a model for grounding spatial descriptors in 3D scenes. Consider inter- preting the instructions: Take the book and put it on the shelf. One critical element of being able to interpret this sentence is associating the refer- ring expression the book with the corresponding object in the world. Another important component of understanding the command above is translat- ing the phrase on the shelf to a location in space. We call such phrases spatial descriptors. While spatial descriptors are closely related to referring expressions, they are distinct in that they can refer to locations even when there is nothing there. An intuitive way to model this is to reason over spatial regions as first-class entities, rather than taking an object-centric approach. Following a long tradition of using game envi- ronments for AI, we adopt Minecraft as the setting for our work. Minecraft has previously been used Misty is hanging in the air next to the wooden shelf with the plant on it. (a) (b) Figure 1: An example from our dataset. (a) The Minecraft scene and its natural language description. (b) Given the choice between six possible locations, our model assigns the highest probability to the location consistent with the natural language description. for work on planning and navigation (Oh et al., 2016; Tessler et al., 2016), and we expand on this by using it for grounded language understanding. As a sandbox game, it can be used to construct a wide variety of environments that capture many interesting aspects of the real world. At the same time, it is easy to extract machine-interpretable representations from the game. We construct a dataset of Minecraft scenes with natural-language annotations, and propose a task that evaluates understanding spatial descriptors. Our task is formulated in terms of locating a pink, cube-shaped character named Misty given a scene, a natural language description, and a set of loca- tions to choose from. An example from our dataset is shown in Figure 1. The Minecraft scene repre- sentation …

Content truncated to 500 words