Image Retrieval from Contextual Descriptions

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3426 - 3440 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Image Retrieval from Contextual Descriptions Benno Krojer1 Vaibhav Adlakha1 Vibhav Vineet3 Yash Goyal4 Edoardo Ponti1 Siva Reddy1,2 1Mila/McGill University 2Facebook CIFAR AI Chair 3Microsoft Research 4Samsung - SAIT AI Lab, Montreal benno.krojer@mila.quebec siva.reddy@mila.quebec Abstract The ability to integrate context, including per- ceptual and temporal cues, plays a pivotal role in grounding the meaning of a linguistic utter- ance. In order to measure to what extent cur- rent vision-and-language models master this ability, we propose a new multimodal chal- lenge, Image Retrieval from Contextual De- scriptions (IMAGECODE). In particular, mod- els are tasked with retrieving the correct im- age from a set of 10 minimally contrastive candidates based on a contextual description. As such, each description contains only the details that help distinguish between images. Because of this, descriptions tend to be com- plex in terms of syntax and discourse and re- quire drawing pragmatic inferences. Images are sourced from both static pictures and video frames. We benchmark several state-of-the-art models, including both cross-encoders such as ViLBERT and bi-encoders such as CLIP, on IMAGECODE. Our results reveal that these models dramatically lag behind human perfor- mance: the best variant achieves an accuracy of 20.9 on video frames and 59.4 on static pic- tures, compared with 90.8 in humans. Further- more, we experiment with new model variants that are better equipped to incorporate visual and temporal context into their representations, which achieve modest gains. Our hope is that IMAGECODE will foster progress in grounded language understanding by encouraging mod- els to focus on fine-grained visual differences. We make code and dataset publicly available.1 1 Introduction Natural languages are highly contextual (Fodor, 2001): for a listener, recovering the speaker’s in- tended meaning requires integrating information from different streams, such as grounding in per- ception (Pecher and Zwaan, 2005), shared world knowledge, and temporal reasoning (Wilson and Sperber, 1998). These processes, more generally, 1https://github.com/McGill-NLP/imagecode (a) Frame 1 (b) Frame 2 (c) Frame 3 (d) Frame 4 Figure 1: An example of the new challenge, Image Retrieval from Contextual Descriptions (IMAGECODE): “The girl in blue is to the left of the girl in the middle with the purple shoes. The girl in blue is not obscured in any way.” Frames 5–10 are left out for simplicity’s sake. The target image, frame 3, is in green, whereas the incorrect frames are in red. fall under the umbrella term of pragmatics (Grice, 1957). Despite recent progress in multimodal sys- tems, it remains unclear to which extent they can handle settings where context plays a major role, such as in real-world communication. To this end, we present a new challenge that requires multimodal models to leverage context to retrieve images from text. In particular, given a contextual description and a set of minimally contrastive candidate images, i.e. differing only in some details, the model has to retrieve the tar- get …

Content truncated to 500 words