How “Multi” is Multi-Document Summarization?

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 582 - 601 December 7-11, 2022 ©2022 Association for Computational Linguistics Abstract Visual Reasoning with Tangram Shapes Anya Ji1, Noriyuki Kojima1∗, Noah Rush1∗, Alane Suhr1,3∗, Wai Keen Vong2, Robert D. Hawkins4, and Yoav Artzi1 1Cornell University 2New York University 3Allen Institute for AI 4Princeton University {aj592, nk654}@cornell.edu noahjrush@gmail.com waikeen.vong@nyu.edu rdhawkins@princeton.edusuhr@cs.cornell.edu yoav@cs.cornell.edu Abstract We introduce KILOGRAM, a resource for study- ing abstract visual reasoning in humans and machines. Drawing on the history of tangram puzzles as stimuli in cognitive science, we build a richly annotated dataset that, with >1k dis- tinct stimuli, is orders of magnitude larger and more diverse than prior resources. It is both vi- sually and linguistically richer, moving beyond whole shape descriptions to include segmen- tation maps and part labels. We use this re- source to evaluate the abstract visual reasoning capacities of recent multi-modal models. We observe that pre-trained weights demonstrate limited abstract reasoning, which dramatically improves with fine-tuning. We also observe that explicitly describing parts aids abstract rea- soning for both humans and models, especially when jointly encoding the linguistic and visual inputs. 1 Introduction Reference is a core function of natural language that relies on shared conventions and visual con- cepts. For example, in English, a speaker may use the term dog to refer to a particular animal of the species canis familiaris, or, through abstraction, to an object with a less strongly conventionalized name, such as the shape at the top of Figure 1. A speaker might refer to such a shape as looking like a dog, and even point to its parts, like its head and tail, despite having few visual features in common with the ordinary referent. Comprehension and generation of references are critical for systems to engage in natural language interaction, and have been studied extensively with focus on ordinary references (e.g., Viethen and Dale, 2008; Mitchell et al., 2010; FitzGerald et al., 2013; Mao et al., 2016; Yu et al., 2016), in con- trast to the visual abstraction illustrated in Figure 1. *Equal contribution, alphabetically ordered. bird head beak head cloaked woman in wind plume cloak hair blowing hood chest dog dog front leg back leg curly tail body body leg tail head head Figure 1: Two example tangrams, each with two dif- ferent annotations. Each annotation includes a whole- shape description (bold), segmentation to parts (in color), and naming of parts (linked to each part). The top example shows low variability with near-perfect agreement, while the bottom shows high variability with divergence of language and segmentation. We address this gap by adopting an influential paradigm for probing human coordination in the cognitive science literature: reference games with abstract tangram shapes (e.g. Clark and Wilkes- Gibbs, 1986; Fox Tree, 1999; Hawkins et al., 2020). Unlike photographs of natural objects, where there is often a single canonical label, tangrams are fundamentally ambiguous. While some shapes fall under strong existing conventions and elicit consen- sus about appropriate names …

Content truncated to 500 words