Paparazzi: A Deep Dive into the Capabilities of Language and Vision Models for Grounding Viewpoint Descriptions

Findings of the Association for Computational Linguistics: EACL 2023, pages 828–843 May 2-6, 2023 ©2023 Association for Computational Linguistics Paparazzi: A Deep Dive into the Capabilities of Language and Vision Models for Grounding Viewpoint Descriptions Henrik Voigt1, Jan Hombeck1, Monique Meuschke3, Kai Lawonn1 and Sina Zarrieß2 1University of Jena 2University of Bielefeld 3University of Magdeburg 1first.last@uni-jena.de 2first.last@uni-bielefeld.de 3last@isg.cs.uni-magdeburg.de Abstract Existing language and vision models achieve impressive performance in image-text under- standing. Yet, it is an open question to what extent they can be used for language under- standing in 3D environments and whether they implicitly acquire 3D object knowledge, e.g. about different views of an object. In this pa- per, we investigate whether a state-of-the-art language and vision model, CLIP, is able to ground perspective descriptions of a 3D object and identify canonical views of common ob- jects based on text queries. We present an eval- uation framework that uses a circling camera around a 3D object to generate images from dif- ferent viewpoints and evaluate them in terms of their similarity to natural language descriptions. We find that a pre-trained CLIP model performs poorly on most canonical views and that fine- tuning using hard negative sampling and ran- dom contrasting yields good results even under conditions with little available training data. 1 Introduction Recent advancements in pre-training large-scale language and vision (L&V) models, such as CLIP (Radford et al., 2021), have led to exceptional performance on benchmarks and leaderboards in 2D image-text retrieval (Shen et al., 2021; Fang et al., 2021; Baldrati et al., 2022). However, the image-text data in these benchmarks have specific properties and biases (Thomason et al., 2022) that may limit the language grounding capabilities of existing L&V models and their robustness in real- word scenarios (Khandelwal et al., 2022; Gadre et al., 2022). A fundamental bias in existing L&V data comes from the fact that images generally show single, human-centric views of different ob- jects. This raises a simple but intriguing ques- tion: to what extent can a model acquire knowl- edge about the concept of viewpoints and identify different views on the same object? Figure 1 il- lustrates this challenge, showing the top-3 images Figure 1: Top-3 retrieval results for car/airplane from the bottom using CLIP on the LAION-5B dataset.2 retrieved by CLIP for two basic viewpoint descrip- tions, car/airplane from the bottom, in the LAION- 5B (Schuhmann et al., 2021) data set: the airplane images mostly correspond to the correct view, but none of the car images shows a bottom view. It suggests that the model does not generalize the meaning of viewpoint descriptions across different objects,1 and may fail to acquire visual-linguistic knowledge that would be needed in more realistic 3D scenarios, such as when instructing a drone to take a picture of an object from a specific view- point (Thomason et al., 2020; Fan et al., 2022). This opens the door for a systematic examination of the capabilities of L&V models for grounding viewpoint descriptions, delving into the question of why, …

Content truncated to 500 words