CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6495–6513 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6495 CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication Jin-Hwa Kim∗ SK T-Brain† jnhwkim@sktbrain.com Nikita Kitaev∗ University of California, Berkeley† kitaev@cs.berkeley.edu Xinlei Chen, Marcus Rohrbach Facebook AI Research {xinleic,mrf}@fb.com Byoung-Tak Zhang Seoul National University btzhang@bi.snu.ac.kr Yuandong Tian Facebook AI Research yuandong@fb.com Dhruv Batra & Devi Parikh Georgia Institute of Technology, Facebook AI Research {parikh,dbatra}@gatech.edu Abstract In this work, we propose a goal-driven collab- orative task that combines language, percep- tion, and action. Specifically, we develop a Collaborative image-Drawing game between two agents, called CoDraw. Our game is grounded in a virtual world that contains mov- able clip art objects. The game involves two players: a Teller and a Drawer. The Teller sees an abstract scene containing multiple clip art pieces in a semantically meaningful config- uration, while the Drawer tries to reconstruct the scene on an empty canvas using avail- able clip art pieces. The two players com- municate with each other using natural lan- guage. We collect the CoDraw dataset of ∼10K dialogs consisting of ∼138K messages exchanged between human players. We de- fine protocols and metrics to evaluate learned agents in this testbed, highlighting the need for a novel crosstalk evaluation condition which pairs agents trained independently on disjoint subsets of the training data. We present models for our task and benchmark them using both fully automated evaluation and by having them play the game live with humans. 1 Introduction Building agents that can interact with humans in natural language while perceiving and taking ac- tions in their environments is one of the fundamen- tal goals in artificial intelligence. To this end, it will be necessary to ground language into percep- tion and action (Harnad, 1990; Barsalou, 1999), ∗ The first two authors contributed equally to this work. † Work performed while the authors were interns at Facebook AI Research. a. Teller View Drawing Canvas b. Drawer View 1. Drawer: Ready 1. You: There’s a girl on the left side. SEND Target Image Chat Box 2. Drawer: What does the girl look like? 2. You: She is holding a beach ball in the arm that is up. 1. Teller: There’s a girl on the left side. 2. You: What does the girl look like? SEND Chat Box Done 2. Teller: She is holding a beach ball in the arm that is up. Drag & Drop waiting for Drawer’s message… 1. You: Ready Figure 1: Overview of the proposed Collaborative Drawing (CoDraw) task. The game consists of two players – Teller and Drawer. The Teller sees an abstract scene, while the Drawer sees an initially empty canvas. Both players need to collaborate and communicate so that the Drawer can drag and drop clip art objects to reconstruct the target scene that is only visible to the Teller. where, e.g., nouns are connected to percepts and verbs relate to actions taken …

Content truncated to 500 words