VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8253 - 8280 May 22-27, 2022 c⃝2022 Association for Computational Linguistics VALSE : A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena Letitia Parcalabescu1∗ Michele Cafagna2 Lilitta Muradjan1 Anette Frank1 Iacer Calixto3,4 Albert Gatt5,2 1Heidelberg University, Department of Computational Linguistics 2University of Malta, Institute of Linguistics and Language Technology 3New York University 4ILLC, University of Amsterdam 5Utrecht University, Department of Information and Computing Sciences Abstract We propose VALSE (Vision And Language Structured Evaluation), a novel benchmark de- signed for testing general-purpose pretrained vision and language (V&L) models for their visio-linguistic grounding capabilities on spe- cific linguistic phenomena. VALSE offers a suite of six tests covering various linguis- tic constructs. Solving these requires models to ground linguistic phenomena in the visual modality, allowing more fine-grained evalua- tions than hitherto possible. We build VALSE using methods that support the construction of valid foils, and report results from evaluating five widely-used V&L models. Our experi- ments suggest that current models have consid- erable difficulty addressing most phenomena. Hence, we expect VALSE to serve as an impor- tant benchmark to measure future progress of pretrained V&L models from a linguistic per- spective, complementing the canonical task- centred V&L evaluations. 1 Introduction General-purpose pretrained vision and language (V&L) models have gained notable performance on many V&L tasks (Lu et al., 2019; Tan and Bansal, 2019; Li et al., 2019; Chen et al., 2020; Li et al., 2020a; Su et al., 2020). As a result, V&L research has changed its focus from task-specific architec- tures to fine-tuning large V&L models. Current benchmarks give a good perspective on model performance on a wide range of V&L tasks (Cao et al., 2020; Lourie et al., 2021; Li et al., 2021), but the field is only starting to assess why models perform so well and whether models learn specific capabilities that span multiple V&L tasks. Specifically, we lack detailed understand- ing of the extent to which such models are able to ground linguistic phenomena—from morphosyn- tax to semantics—in the visual modality (Bernardi ∗ Corresponding author parcalabescu@cl. uni-heidelberg.de. and Pezzelle, 2021). For example, recent evidence suggests that models are insensitive to linguistic distinctions of verb-argument structure (Hendricks and Nematzadeh, 2021) and word order (Cirik et al., 2018; Akula et al., 2020). Our work addresses this gap with VALSE (Vi- sion And Language Structured Evaluation), a benchmark for V&L model evaluation compris- ing six tasks, or ‘pieces’, where each piece has the same structure: given a visual input, a model is asked to distinguish real captions from foils, where a foil is constructed from a caption by altering a word or phrase that realizes a specific linguistic phenomenon, e.g., semantic number of nouns, verb argument structure, or coreference. VALSE uses a resource-lean diagnostic setup that dispenses with large-scale annotation (e.g., of bounding boxes), and builds on existing high-quality image caption- ing and VQA data. VALSE is designed to lever- age the …

Content truncated to 500 words