A Large-Scale Multilingual Study of Visual Constraints on Linguistic Selection of Descriptions

Findings of the Association for Computational Linguistics: EACL 2023, pages 2285–2299 May 2-6, 2023 ©2023 Association for Computational Linguistics A Large-Scale Multilingual Study of Visual Constraints on Linguistic Selection of Descriptions Uri Berger1,2, Lea Frermann2, Gabriel Stanovsky1, and Omri Abend1 1School of Computer Science and Engineering, The Hebrew University of Jerusalem 2School of Computing and Information Systems, University of Melbourne {uri.berger2, gabriel.stanovsky, omri.abend}@mail.huji.ac.il lea.frermann@unimelb.edu.au Abstract We present a large, multilingual study into how vision constrains linguistic choice, covering four languages and five linguistic properties, such as verb transitivity or use of numerals. We propose a novel method that leverages exist- ing corpora of images with captions written by native speakers, and apply it to nine corpora, comprising 600k images and 3M captions. We study the relation between visual input and lin- guistic choices by training classifiers to predict the probability of expressing a property from raw images, and find evidence supporting the claim that linguistic properties are constrained by visual context across languages. We comple- ment this investigation with a corpus study, tak- ing the test case of numerals. Specifically, we use existing annotations (number or type of ob- jects) to investigate the effect of different visual conditions on the use of numeral expressions in captions, and show that similar patterns emerge across languages. Our methods and findings both confirm and extend existing research in the cognitive literature. We additionally discuss possible applications for language generation. We make our codebase publicly available.1 1 Introduction In recent years, vision and language models have been shown to outperform models trained on a sin- gle modality in a variety of domains, such as lan- guage modeling (Ororbia et al., 2019), document quality assessment (Shen et al., 2020), and visual classification and segmentation (Frome et al., 2013; Radford et al., 2021; Berger et al., 2022). While empirical evidence exists, including from the studies cited above, that each modality can ben- efit the other for task performance, less attention has been devoted to the broader question of how the two modalities influence and constrain one an- other. In this study, we focus on one aspect of this question: How does vision constrain language? 1github.com/SLAB-NLP/visual_constraints_on_descriptions Figure 1: A demonstration of how visual cues in im- ages may constrain linguistic choices in their captions. The image on the left in which the agent is visible is described using an active voice “A man... is throwing a child... in the air”, while in the right image the agent is not visible and the annotator chose a passive construc- tion: “A little boy... is thrown in the air”. Images and captions taken from Flickr30k. We study the relation between the semantic con- tent of an image and the language used to describe it. As an example, consider Figure 1. Captions of the right image, which crops the agent, use passive voice more frequently than those of the left (taken from Flickr30k, Young et al., 2014). We aim to study such trends by examining the influence of vi- sual features …

Content truncated to 500 words