Learning to execute instructions in a Minecraft dialogue

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2589–2602 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2589 Learning to execute instructions in a Minecraft dialogue Prashant Jayannavar Anjali Narayan-Chen University of Illinois at Urbana-Champaign {paj3, nrynchn2, juliahmr}@illinois.edu Julia Hockenmaier Abstract The Minecraft Collaborative Building Task is a two-player game in which an Architect A in- structs a Builder B to construct a target struc- ture out of 3D blocks. We consider the task of predicting B’s action sequences (block place- ments and removals) in a given game context, and show that capturing B’s past actions as well as B’s perspective leads to a significant improvement in performance on this challeng- ing language understanding problem. 1 Introduction There is a long-standing interest in building in- teractive agents that can communicate with hu- mans about and operate within the physical world (e.g. Winograd (1971)). The goal for agents in this scenario is to not only be able to engage in rich natural language discourse with their human con- versation partners, but also to ground that discourse to physical objects, and execute instructions in the real world. Traditional dialogue scenarios are ei- ther completely ungrounded (Ritter et al., 2010; Schrading et al., 2015), focus on slot-value filling tasks (Kim et al., 2016b,a; Budzianowski et al., 2018) which instead require grounding to entities in a knowledge base, or operate within static en- vironments, such as images (Das et al., 2017) or videos (Pasunuru and Bansal, 2018). Relevant ef- forts in robotics have largely focused on single-shot instruction following, and are mostly constrained to simple language (Roy and Reiter, 2005; Tellex et al., 2011) with limited resources (Thomason et al., 2015; Misra et al., 2016; Chai et al., 2018). The recently introduced Minecraft Collaborative Building Task and the corresponding Minecraft Dialogue Corpus (Narayan-Chen et al., 2019) is one attempt to bridge this gap within the simulated game world of Minecraft. In this task, two players, an Architect (A) instructs a Builder (B) to con- struct a target structure out of multi-colored build- ing blocks. The corpus consists of 509 game logs between humans that perform this task. Narayan- Chen et al. (2019) focus on generating Architect utterances. In this paper, we explore models for building an automated Builder agent.1 We focus on the subtask of predicting the Builder’s block placements, and leave the back-and-forth dialogue aspect of the overall task required of a fully inter- active Builder agent to future work. We define the Builder Action Prediction (BAP) task in Section 2, describe our models in Section 3, an approach to augment the training data in Section 4, and our experiments in Section 5. We analyze results and highlight challenges of the BAP task in Section 6. 2 Dataset and Task 2.1 The Minecraft Dialogue Corpus The Minecraft Dialogue Corpus (Narayan-Chen et al., 2019) consists of 509 human-human dia- logues and game logs for the Minecraft Collabora- tive Building Task, a two-player game in a simu- lated …

Content truncated to 500 words