Ranking-Enhanced Unsupervised Sentence Representation Learning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1559–1582 July 9-14, 2023 ©2023 Association for Computational Linguistics Measuring Progress in Fine-grained Vision-and-Language Understanding Emanuele Bugliarello∗,D,C Laurent SartranD Aishwarya AgrawalD Lisa Anne Hendricks‡,D Aida Nematzadeh‡,D DDeepMind CUniversity of Copenhagen Abstract While pretraining on large-scale image–text data from the Web has facilitated rapid progress on many vision-and-language (V&L) tasks, re- cent work has demonstrated that pretrained models lack “fine-grained” understanding, such as the ability to recognise relationships, verbs, and numbers in images. This has resulted in an increased interest in the community to either develop new benchmarks or models for such capabilities. To better understand and quantify progress in this direction, we investigate four competitive V&L models on four fine-grained benchmarks. Through our analysis, we find that X-VLM (Zeng et al., 2022) consistently out- performs other baselines, and that modelling innovations can impact performance more than scaling Web data, which even degrades perfor- mance sometimes. Through a deeper investiga- tion of X-VLM, we highlight the importance of both novel losses and rich data sources for learning fine-grained skills. Finally, we inspect training dynamics, and discover that for some tasks, performance peaks early in training or significantly fluctuates, never converging. 1 Introduction Fine-grained multimodal skills (e.g., understanding relationships and recognising verbs) require identi- fying and relating various entities across both im- age and text modalities. Vision-and-language mod- els (VLMs) need such skills to robustly perform well on real-world vision-and-language (V&L) ap- plications; e.g., a coarse-grained model tested on image retrieval to “find an image where something is on a sofa” might incorrectly return an image of a cat sitting below the sofa. As another example, in captioning, a model might incorrectly describe an image where “someone is selling a sweater” as “someone is buying a sweater,” if it does not have a precise understanding of the two verbs. ∗Work completed during an internship at DeepMind. ‡denotes equal senior contribution. Correspondence to: Emanuele Bugliarello <emanuele@di.ku.dk>. However, common V&L benchmarks (e.g., Lin et al., 2014; Goyal et al., 2017; Suhr et al., 2019) do not explicitly shed light on such fine-grained un- derstanding. Indeed, in the last few years, there has been an increase in the number of bench- marks which demonstrate that current, coarse- grained models struggle with fine-grained under- standing (Hendricks and Nematzadeh, 2021; Par- calabescu et al., 2022; Salin et al., 2022; Thrush et al., 2022). Meanwhile, more models have been designed specifically to learn a better mapping be- tween visual and textual modalities (e.g., Yao et al., 2022a,b; Zeng et al., 2022; Gao et al., 2022). While such models perform well on coarse-grained re- trieval and other downstream tasks, they have not been directly evaluated on fine-grained understand- ing. Consequently, it is unclear if the performance gains are due to tighter, more fine-grained repre- sentations introduced by model innovations at the pretraining stage. To fill this gap, we analyse sev- eral recent models with innovations designed for a better image–text alignment and …

Content truncated to 500 words