Collaborative Dialogue in Minecraft

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5405–5415 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5405 Collaborative Dialogue in Minecraft Anjali Narayan-Chen∗ Prashant Jayannavar∗ University of Illinois at Urbana-Champaign {nrynchn2, paj3, juliahmr}@illinois.edu Julia Hockenmaier Abstract We wish to develop interactive agents that can communicate with humans to collaboratively solve tasks in grounded scenarios. Since com- puter games allow us to simulate such tasks without the need for physical robots, we de- fine a Minecraft-based collaborative building task in which one player (A, the Architect) is shown a target structure and needs to instruct the other player (B, the Builder) to build this structure. Both players interact via a chat inter- face. A can observe B but cannot place blocks. We present the Minecraft Dialogue Corpus, a collection of 509 conversations and game logs. As a first step towards our goal of developing fully interactive agents for this task, we con- sider the subtask of Architect utterance gener- ation, and show how challenging it is. 1 Introduction Building interactive agents that can successfully communicate with humans about the physical world around them to collaboratively solve tasks in this environment is a long-sought goal of AI (e.g. Winograd, 1971). Such situated dialogue poses challenges that go beyond what is required for the slot-value filling tasks performed by stan- dard dialogue systems (e.g. Kim et al., 2016, 2017; Budzianowski et al., 2018) or chatbots (e.g. Ritter et al., 2010; Schrading et al., 2015; Lowe et al., 2015), as well as for so-called visual dialogue where users talk about a static image (Das et al., 2017) or video-context dialogue where users inter- act in a chat room while viewing a live-streamed video (Pasunuru and Bansal, 2018). It requires the ability to refer to real-world objects and spatial re- lations that depend on the current position of the speakers as well as changes in the environment. Due to the expense of actual human-robot commu- nication (e.g. Tellex et al., 2011; Thomason et al., ∗Both authors equally contributed to the paper. 2015; Misra et al., 2016; Chai et al., 2018), simu- lated environments that allow easier experimenta- tion are commonly used (Koller et al., 2010; Chen and Mooney, 2011; Janarthanam et al., 2012). In this paper, we therefore introduce the Minecraft Collaborative Building Task, in which pairs of users control avatars in the Minecraft virtual environment and collaboratively build 3D structures in a Blocks World-like scenario while communicating solely via text chat (Section 3). We have built a data collection platform and have used it to collect the Minecraft Dialogue Cor- pus, consisting of 509 human-human written di- alogues, screenshots and complete game logs for this task (Section 4). While our ultimate goal is to develop fully interactive agents that can collab- orate with humans successfully on this task, we first consider the subtask of Architect utterance generation (Section 5) and describe a set of base- line models that encode both the dialogue history …

Content truncated to 500 words