Things not Written in Text: Exploring Spatial Commonsense from Visual Signals

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2365 - 2376 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Things not Written in Text: Exploring Spatial Commonsense from Visual Signals Xiao Liu1, Da Yin2, Yansong Feng1,3∗and Dongyan Zhao1,4,5 1Wangxuan Institute of Computer Technology, Peking University 2Computer Science Department, University of California, Los Angeles 3The MOE Key Laboratory of Computational Linguistics, Peking University 4Artificial Intelligence Institute of Peking University 5State Key Laboratory of Media Convergence Production Technology and Systems {lxlisa,fengyansong,zhaody}@pku.edu.cn da.yin@cs.ucla.edu Abstract Spatial commonsense, the knowledge about spatial position and relationship between ob- jects (like the relative size of a lion and a girl, and the position of a boy relative to a bicycle when cycling), is an important part of commonsense knowledge. Although pre- trained language models (PLMs) succeed in many NLP tasks, they are shown to be in- effective in spatial commonsense reasoning. Starting from the observation that images are more likely to exhibit spatial commonsense than texts, we explore whether models with visual signals learn more spatial commonsense than text-based PLMs. We propose a spatial commonsense benchmark that focuses on the relative scales of objects, and the positional relationship between people and objects un- der different actions. We probe PLMs and models with visual signals, including vision- language pretrained models and image synthe- sis models, on this benchmark, and find that image synthesis models are more capable of learning accurate and consistent spatial knowl- edge than other models. The spatial knowl- edge from image synthesis models also helps in natural language understanding tasks that require spatial commonsense. Code and data are available at https://github.com/ xxxiaol/spatial-commonsense. 1 Introduction Spatial perception, the ability to detect the spa- tial position and to infer the relationship between visual stimuli (Donnon et al., 2005; Saj and Baris- nikov, 2015), is basic but important for human beings (Pellegrino et al., 1984). It is of everyday use, from understanding the surrounding environ- ment, like when seeing a woman sitting in a car with her hands on the steering wheel, we know she is probably driving, to processing spatial infor- mation and performing reasoning, like navigating ∗ Corresponding author. Lion Lion Cycling Cycling Texts Images The lion (Panthera leo) is a large felid of the genus Panthera native to Africa and India. It has a muscular, deep- chested body, short, rounded head, round ears, and a hairy tuft at the end of its tail ... Cycling, also called bicycling or biking, is the use of bicycles for transport, recreation, exercise or sport. People engaged in cycling are referred to as "cyclists", "bicyclists", or "bikers" ... How big is a lion? Where is a boy relative to a bike when cycling? Figure 1: Texts and images related to lion and cycling. Images exhibit more explicit spatial knowledge than texts. through a dense forest. We regard the knowledge needed in spatial perception as spatial common- sense. Humans start to develop spatial perception and acquire spatial commonsense from infancy, and …

Content truncated to 500 words