ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5198 - 5215 May 22-27, 2022 c⃝2022 Association for Computational Linguistics ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension Sanjay Subramanian∗1 Will Merrill2 Trevor Darrell1 Matt Gardner3 Sameer Singh4,5 Anna Rohrbach1 1UC Berkeley 2New York University 3Microsoft Semantic Machines 4UC Irvine 5Allen Institute for AI (AI2) {sanjayss,trevordarrell,anna.rohrbach}@berkeley.edu, wcm9940@nyu.edu, mattgardner@microsoft.com, sameer@uci.edu Abstract Training a referring expression comprehension (ReC) model for a new visual domain requires collecting referring expressions, and potentially corresponding bounding boxes, for images in the domain. While large-scale pre-trained mod- els are useful for image classification across domains, it remains unclear if they can be ap- plied in a zero-shot manner to more complex tasks like ReC. We present ReCLIP, a simple but strong zero-shot baseline that repurposes CLIP, a state-of-the-art large-scale model, for ReC. Motivated by the close connection be- tween ReC and CLIP’s contrastive pre-training objective, the first component of ReCLIP is a region-scoring method that isolates object pro- posals via cropping and blurring, and passes them to CLIP. However, through controlled ex- periments on a synthetic dataset, we find that CLIP is largely incapable of performing spatial reasoning off-the-shelf. Thus, the second com- ponent of ReCLIP is a spatial relation resolver that handles several types of spatial relations. We reduce the gap between zero-shot baselines from prior work and supervised models by as much as 29% on RefCOCOg, and on RefGTA (video game imagery), ReCLIP’s relative im- provement over supervised ReC models trained on real images is 8%. 1 Introduction Visual referring expression comprehension (ReC)— the task of localizing an object in an image given a textual referring expression—has applications in a broad range of visual domains. For example, ReC is useful for guiding a robot in the real world (Shridhar et al., 2020) and also for creating natu- ral language interfaces for software applications with visuals (Wichers et al., 2018). Though the task is the same across domains, the domain shift is problematic for supervised referring expression models, as shown in Figure 1: the same simple ∗This work was done while Sanjay, Will, and Matt were affiliated with AI2. (a) RefCOCO+ (Yu et al., 2016) (b) RefGTA (Tanaka et al., 2019) Figure 1: Predictions from ReCLIP (cyan) and UNITER-Large (Chen et al., 2020) (red) for the same referring expression on images from two visual domains. UNITER-Large fails on the GTA (video game) domain, while ReCLIP selects the correct proposal in both cases. Close-ups of the two GTA boxes are shown. referring expression is localized correctly in the training domain but incorrectly in a new domain. Collecting task-specific data in each domain of interest is expensive. Weakly supervised ReC (Rohrbach et al., 2016) partially addresses this is- sue, since it does not require the ground-truth box for each referring expression, but it still assumes the availability of referring expressions paired with images and trains on these. Given a large-scale pre- trained vision and language model and a method …

Content truncated to 500 words