Beyond Instructional Videos: Probing for More Diverse Visual-Textual Grounding on YouTube

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8812–8822, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8812 Beyond Instructional Videos: Probing for More Diverse Visual-Textual Grounding on YouTube Jack Hessel Zhenhai Zhu Bo Pang Radu Soricut Allen Institute for AI Google Research jackh@allenai.org {zhenhai,bopang,rsoricut}@google.com Abstract Pretraining from unlabelled web videos has quickly become the de-facto means of achiev- ing high performance on many video under- standing tasks. Features are learned via pre- diction of grounded relationships between vi- sual content and automatic speech recogni- tion (ASR) tokens. However, prior pretrain- ing work has been limited to only instruc- tional videos; a priori, we expect this domain to be relatively “easy:” speakers in instruc- tional videos will often reference the literal ob- jects/actions being depicted. We ask: can sim- ilar models be trained on more diverse video corpora? And, if so, what types of videos are “grounded” and what types are not? We fit a representative pretraining model to the di- verse YouTube8M dataset, and study its suc- cess and failure cases. We find that visual- textual grounding is indeed possible across previously unexplored video categories, and that pretraining on a more diverse set results in representations that generalize to both non- instructional and instructional domains. 1 Introduction Self-supervised pretraining approaches have re- cently been adapted to web videos (Sun et al., 2019a,b; Miech et al., 2019, 2020; Zhu and Yang, 2020; Amrani et al., 2020); the resulting mod- els have achieved state-of-the-art performance on a wide range of video understanding tasks, e.g., dense caption generation, action localization, etc. In general, the pretraining step requires a large, unlabelled corpus of web videos. The training ob- jective aligns visual content (i.e., video segments) with automatic speech recognition (ASR) tokens, and the resulting representations are fine-tuned for downstream tasks. The assumption underlying this family of approaches is that, in the pretrain- ing corpus, spoken words have some consistent, grounded relationship with the temporally corre- sponding visual content. However, in contrast to the highly diverse corpora utilized for text-based pretraining (Wikipedia, Common Crawl, etc.), pretraining for web videos (so far) has been limited to instruc- tional videos. This domain restriction is motivated by the commonly accepted notion that “procedu- ral knowledge tends to be inherently multimodal” (Malmaud et al., 2015). We expect that the seman- tic information in video frames and ASR tokens is readily correlated in instructional videos. But corpus diversity brings significant benefits: in the text-only case, models can effectively represent diverse real-world entities (Roberts et al., 2020) precisely because pretraining is not restricted to, e.g., only fictional stories (Zhu et al., 2015). In search of more general representations, our main question is: does video-ASR pretraining “work” for more diverse pretraining corpora? Are certain categories of non-instructional videos “groundable,” thus enabling diverse representa- tion learning? Or are some types too difficult, only acting as training noise? We conclude that: 1) grounding is indeed possible in a wide range of yet-to-be-computationally-exploited YouTube video categories, e.g., walk-throughs, vehicles, …

Content truncated to 500 words