MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1112–1125 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1112 MINDCRAFT: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks Cristian-Paul Bara∗ University of Michigan cpbara@umich.edu Sky CH-Wang∗† Columbia University skywang@cs.columbia.edu Joyce Chai University of Michigan chaijy@umich.edu Abstract An ideal integration of autonomous agents in a human world implies that they are able to col- laborate on human terms. In particular, theory of mind plays an important role in maintain- ing common ground during human collabora- tion and communication. To enable theory of mind modeling in situated interactions, we in- troduce a fine-grained dataset of collaborative tasks performed by pairs of human subjects in the 3D virtual blocks world of Minecraft. It provides information that captures partners’ beliefs of the world and of each other as an in- teraction unfolds, bringing abundant opportu- nities to study human collaborative behaviors in situated language communication. As a first step towards our goal of developing embodied AI agents able to infer belief states of collabo- rative partners in situ, we build and present re- sults on computational models for several the- ory of mind tasks. 1 Introduction Creating embodied, situated agents able to move in, communicate naturally about, and collaborate on human terms in the physical world has been a persisting goal in artificial intelligence (Winograd, 1972). During communication in such a setting, agents not only need to ground entities in language to that of the physical world; efficient and accurate human-agent collaboration further requires agents to reason about the progress of the task at hand and to plan and execute a series of collaborative steps, whilst maintaining common ground (Clark, 1996) with collaboration partners, in order to achieve a certain goal. Despite recent advances, we are still far away from fully enabling these desired agent behaviors. ∗Equal Contribution. †Work performed while the author was an undergraduate research assistant at the University of Michigan. One key challenge is in an agent’s ability to estab- lish and maintain common ground in tandem with human partners, especially in a setting where be- liefs about the world and of each other may change on the fly (Popat and Palmer, 2005; Powers et al., 2005). It is important to understand how changes in a dynamic, physical world affect agents’ be- liefs of each other—i.e., theory of mind (Premack and Woodruff, 1978)—and how such beliefs influ- ence teamwork and communication in collaborative tasks. As a first step to address this question, this paper explores theory of mind modeling (Chan- drasekaran et al., 2017; Rabinowitz et al., 2018; Jara-Ettinger, 2019) in situated language communi- cation (Iwahashi et al., 2009; McGuire et al., 2002) for collaborative tasks within the 3D virtual blocks world of Minecraft. Through a novel experimental setup, we collect a situated dialogue dataset that demonstrates how collaborative partners with a set of asymmetric knowledge and skills are able to col- laborate to achieve joint goals, and how, in particu- lar, their beliefs of …

Content truncated to 500 words