Exposing the Limits of Video-Text Models through Contrast Sets

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3574 - 3586 July 10-15, 2022 ©2022 Association for Computational Linguistics Exposing the Limits of Video-Text Models through Contrast Sets Jae Sung Park1 Sheng Shen2 Ali Farhadi1 Trevor Darrell2 Yejin Choi1,3 Anna Rohrbach2 1 Paul G. Allen School of Computer Science & Engineering, University of Washington 2 University of California, Berkeley 3 Allen Institute for Artificial Intelligence {jspark96, ali, yejin}@cs.washington.edu {sheng.s, trevordarrell, anna.rohrbach}@berkeley.edu Abstract Recent video-text models can retrieve relevant videos based on text with a high accuracy, but to what extent do they comprehend the seman- tics of the text? Can they discriminate between similar entities and actions? To answer this, we propose an evaluation framework that probes video-text models with hard negatives. We au- tomatically build contrast sets, where true tex- tual descriptions are manipulated in ways that change their semantics while maintaining plau- sibility. Specifically, we leverage a pre-trained language model and a set of heuristics to cre- ate verb and person entity focused contrast sets. We apply these in the multiple choice video-to- text classification setting. We test the robust- ness of recent methods on the proposed auto- matic contrast sets, and compare them to ad- ditionally collected human-generated counter- parts, to assess their effectiveness. We see that model performance suffers across all methods, erasing the gap between recent CLIP-based methods vs. the earlier methods. 1 1 Introduction Relating video and text modalities is one of the important goals in vision and language. Video is a complex signal where people and objects act and interact with each other through space and time. Thus correctly associating a textual description and a video requires understanding of entities, their actions and much more, making it a hard problem. One of the popular ways of training and evaluat- ing video-text models is via cross-modal matching. Often the task is formulated as a retrieval problem, where the goal is to select the correct match among many (e.g. thousand) candidates, and distractors are picked randomly (Yu et al., 2018). Another way is via multiple-choice prediction, where the goal is to pick the true match out of several (e.g. 5) candidates (Torabi et al., 2016). The latter allows 1Code is available in https://github.com/ jamespark3922/video-lang-contrast-set Figure 1: Samples of our video-to-text tasks on the MSR-VTT (Xu et al., 2016) and LSMDC dataset (Rohrbach et al., 2017; Park et al., 2020). A hard negative option is added by manipulating verb (top) and entity (bottom) in the ground truth sentence. Two SOTA methods MMT (Gabeur et al., 2020) and CLIP4CLIP (Luo et al., 2021) incorrectly choose the manipulated sentence (option B) in both these cases. for more controlled choice of negatives, which are typically selected from other videos. Commonly, the retrieval setting is used during training to avoid capturing any specific multiple-choice patterns or biases, while both are used for evaluation. Recent methods that leverage the large-scale CLIP model (Radford et al., 2021) show significant improvement …

Content truncated to 500 words