Iconary: A Pictionary-Based Game for Testing Multimodal Communication with Drawings and Text

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1864–1886 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1864 Iconary: A Pictionary-Based Game for Testing Multimodal Communication with Drawings and Text Christopher Clark∗ Jordi Salvador∗ Dustin Schwenk∗ Derrick Bonafilia∗ Mark Yatskar† Eric Kolve∗ Alvaro Herrasti∗ Jonghyun Choi∗Γ Sachin Mehta+ Sam Skjonsberg∗ Carissa Schoenick∗ Aaron Sarnat∗ Hannaneh Hajishirzi∗+ Aniruddha Kembhavi∗+ Oren Etzioni∗ Ali Farhadi∗+ ∗Allen Institute for AI +University of Washington †University of Pennsylvania ΓGwangju Institute of Science and Technology chrisc@allenai.org Abstract Communicating with humans is challenging for AIs because it requires a shared under- standing of the world, complex semantics (e.g., metaphors or analogies), and at times multi- modal gestures (e.g., pointing with a finger, or an arrow in a diagram). We investigate these challenges in the context of Iconary, a collab- orative game of drawing and guessing based on Pictionary, that poses a novel challenge for the research community. In Iconary, a Guesser tries to identify a phrase that a Drawer is draw- ing by composing icons, and the Drawer itera- tively revises the drawing to help the Guesser in response. This back-and-forth often uses canonical scenes, visual metaphor, or icon compositions to express challenging words, making it an ideal test for mixing language and visual/symbolic communication in AI. We pro- pose models to play Iconary and train them on over 55,000 games between human play- ers. Our models are skillful players and are able to employ world knowledge in language models to play with words unseen during train- ing. Elite human players outperform our mod- els, particularly at the drawing task, leaving an important gap for future research to ad- dress. We release our dataset, code, and evalu- ation setup as a challenge to the community at github.com/allenai/iconary. 1 Introduction Communicating with humans is a long-standing goal in AI, and has been studied in the context of natural language for decades. Many of the key challenges in this task, such as using a shared un- derstanding of the world, commonsense reasoning, and metaphor are, however, not language-specific, but are instead general-purpose tools that humans use when communicating through other modalities as well. For example, understanding what means in a text conversation requires grasping metaphor (it is unlikely to be literally suggesting one should put on a party hat), or understanding a sign with a truck swerving requires common-sense reasoning (the intent is to show slippery conditions, not to sug- gest drivers ought to begin swerving themselves). Humans can easily adapt to these different modal- ities, as well as use visual/symbolic tools (e.g., pointing with a finger, or an arrow in a diagram) that cannot be used in a text-only context. To build and test AIs for this skill, we introduce the first task and large-scale dataset for multimodal communi- cation by creating Iconary, a game of drawing and guessing based on Pictionary, along with a dataset of games with human players, proposing automatic and online game playing metrics, and constructing proficient Iconary AIs. In …

Content truncated to 500 words