Executing Instructions in Situated Collaborative Interactions

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2119–2130, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2119 Executing Instructions in Situated Collaborative Interactions Alane Suhr Cornell University suhr@cs.cornell.edu Claudia Yan IBM claudiab.yan@gmail.com Jacob Schluger∗ Cornell University jes543@cornell.edu Stanley Yu∗ Columbia University stanley.yu@columbia.edu Hadi Khader∗∗ Intel hadi.kh.khader@gmail.com Marwa Mouallem∗∗ IBM marwamouallem@gmail.com Iris Zhang Facebook irisz@fb.com Yoav Artzi Cornell University yoav@cs.cornell.edu Abstract We study a collaborative scenario where a user not only instructs a system to complete tasks, but also acts alongside it. This allows the user to adapt to the system abilities by changing their language or deciding to simply accom- plish some tasks themselves, and requires the system to effectively recover from errors as the user strategically assigns it new goals. We build a game environment to study this sce- nario, and learn to map user instructions to system actions. We introduce a learning ap- proach focused on recovery from cascading er- rors between instructions, and modeling meth- ods to explicitly reason about instructions with multiple goals. We evaluate with a new evalu- ation protocol using recorded interactions and online games with human users, and observe how users adapt to the system abilities. 1 Introduction Sequential instruction scenarios commonly as- sume only the system performs actions, and there- fore only its behavior influences the world state. This ignores the collaborative potential of such interactive scenarios and the challenges it intro- duces. When the user acts in the world as well, they can adapt to the system abilities not only by adopting simpler language, but also by deciding to accomplish tasks themselves. The system must then recover from errors as new instructions arrive and be robust to changes in the environment that are not a result of its own actions. In this paper, we introduce CEREALBAR, a collaborative game with natural language instruc- tion, and design modeling, learning, and evalua- tion methods for the problem of sequential instruc- tion following in collaborative interactions. In CE- REALBAR, two agents, a leader and a follower, ∗,∗∗: Equal contribution. All work done at Cornell. Follower’s view Leader’s view Leader Follower . . . ¯x3: turn left and head toward the yellow hearts, but don’t pick them up yet. I’ll get the next card first. ¯x4: Okay, pick up yellow hearts and run past me toward the bush sticking out, on the opposite side is 3 green stars [Set made. New score: 4] . . . Figure 1: A snapshot from an interaction in CEREAL- BAR. The current instruction is in bold. The large im- age shows the entire environment. This overhead view is available only to the leader. The follower sees a first- person view only (bottom right). The zoom boxes (top) show the leader and follower. move in a 3D environment and collect valid sets of cards to earn points. A valid set is a set of three cards with distinct color, shape, and …

Content truncated to 500 words