Visual Spatial Reasoning

Visual Spatial Reasoning Fangyu Liu Guy Emerson Nigel Collier University of Cambridge, UK {fl399, gete2, nhc30}@cam.ac.uk Abstract Spatial relations are a basic part of human cog- nition. However, they are expressed in natural language in a variety of ways, and previous work has suggested that current vision-and- language models (VLMs) struggle to capture relational information. In this paper, we pres- ent Visual Spatial Reasoning (VSR), a dataset containing more than 10k natural text-image pairs with 66 types of spatial relations in En- glish (e.g., under, in front of, facing). While using a seemingly simple annotation format, we show how the dataset includes challenging linguistic phenomena, such as varying refer- ence frames. We demonstrate a large gap be- tween human and model performance: The human ceiling is above 95%, while state-of- the-art models only achieve around 70%. We observe that VLMs’ by-relation performances have little correlation with the number of training examples and the tested models are in general incapable of recognising relations concerning the orientations of objects.1 1 Introduction Multimodal NLP research has developed rap- idly in recent years, with substantial performance gains on tasks such as visual question answering (VQA) (Antol et al., 2015; Johnson et al., 2017; Goyal et al., 2017; Hudson and Manning, 2019; Zellers et al., 2019), vision-language reasoning or entailment (Suhr et al., 2017, 2019; Xie et al., 2019; Liu et al., 2021), and referring expression comprehension (Yu et al., 2016; Liu et al., 2019). Existing benchmarks, such as NLVR2 (Suhr et al., 2019) and VQA (Goyal et al., 2017), define ge- neric paradigms for testing vision-language mod- els (VLMs). However, as we further discuss in § 2, these benchmarks are not ideal for prob- ing VLMs as they typically conflate multiple sources of error and do not allow controlled analysis on specific linguistic or cognitive prop- erties, making it difficult to categorize and fully 1Data and code: github.com/cambridgeltl/visual -spatial-reasoning. understand the model failures. In particular, spa- tial reasoning has been found to be particularly challenging for current models, and much more challenging than capturing properties of individ- ual entities (Kuhnle et al., 2018; Cirik et al., 2018; Akula et al., 2020), even for state-of-the- art models such as CLIP (Radford et al., 2021; Subramanian et al., 2022). Another line of work generates synthetic data- sets in a controlled manner to target specific relations and properties when testing VLMs, e.g., CLEVR (Liu et al., 2019) and ShapeWorld (Kuhnle and Copestake, 2018). However, syn- thetic datasets may accidentally overlook chal- lenges (such as orientations of objects which we will discuss in § 5), and using natural images allows us to explore a wider range of language use. To address the lack of probing evaluation benchmarks in this field, we present VSR (Vi- sual Spatial Reasoning), a controlled dataset that explicitly tests VLMs for spatial reasoning. We choose spatial reasoning as the focus because it is one of the most fundamental capabilities for both humans and VLMs. Such relations are cru- cial to how humans organize their …

Content truncated to 500 words