ArraMon: A Joint Navigation-Assembly Instruction Interpretation Task in Dynamic Environments

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3910–3927 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3910 ARRAMON: A Joint Navigation-Assembly Instruction Interpretation Task in Dynamic Environments Hyounghun Kim Abhay Zala Graham Burri Hao Tan Mohit Bansal UNC Chapel Hill {hyounghk, aszala, ghburri, airsplay, mbansal}@cs.unc.edu Abstract For embodied agents, navigation is an impor- tant ability but not an isolated goal. Agents are also expected to perform specific tasks af- ter reaching the target location, such as pick- ing up objects and assembling them into a par- ticular arrangement. We combine Vision-and- Language Navigation, assembling of collected objects, and object referring expression com- prehension, to create a novel joint navigation- and-assembly task, named ARRAMON. Dur- ing this task, the agent (similar to a Pok´eMON GO player) is asked to find and collect dif- ferent target objects one-by-one by navigating based on natural language (English) instruc- tions in a complex, realistic outdoor environ- ment, but then also ARRAnge the collected ob- jects part-by-part in an egocentric grid-layout environment. To support this task, we imple- ment a 3D dynamic environment simulator and collect a dataset with human-written naviga- tion and assembling instructions, and the cor- responding ground truth trajectories. We also filter the collected instructions via a verifica- tion stage, leading to a total of 7.7K task in- stances (30.8K instructions and paths). We present results for several baseline models (in- tegrated and biased) and metrics (nDTW, CTC, rPOD, and PTC), and the large model-human performance gap demonstrates that our task is challenging and presents a wide scope for fu- ture work.1 1 Introduction Navigation guided via flexible natural language (NL) instructions is a crucial capability for robotic and embodied agents. Such systems should be capable of interpreting human instructions to cor- rectly navigate realistic complex environments and reach destinations by understanding the environ- ment, and associating referring expressions in the 1Our dataset, simulator, and code are publicly available at: https://github.com/hyounghk/ArraMon Navigation Phase: Turn around to face the speed limit sign. Go to the sign and then turn right around the corner. Go to the booth and a little past it and to the right there is a brown hourglass. Pick it up. Assembly Phase: Place the hourglass to the right of the red mug in front of you. TURN 1 TURN 2 Navigation Phase: Turn left to face the dumpster. Go around the building corner, and past the phone booth to the next intersection. At the intersection turn left. Next to the yellow building there is a green bucket. Pick the bucket up. Assembly Phase: Turn right and place the bucket in front of the striped red mug. Figure 1: Navigation and assembly phases (2 turns), via NL (English) instructions in a dynamic 3D envi- ronment. In the navigation phase, agents are asked to find and collect a target object. In the assembly phase, agents have to egocentrically place the collected object at a relative location (navigation turn 2 starts where turn 1 ends; we only show …

Content truncated to 500 words