Text-Derived Knowledge Helps Vision: A Simple Cross-modal Distillation for Video-based Action Anticipation

Findings of the Association for Computational Linguistics: EACL 2023, pages 1882–1897 May 2-6, 2023 ©2023 Association for Computational Linguistics Text-Derived Knowledge Helps Vision: A Simple Cross-modal Distillation for Video-based Action Anticipation Sayontan Ghosh1 Tanvi Aggarwal1 Minh Hoai1 Niranjan Balasubramanian1 1Stony Brook University {sagghosh, taggarwal, minhhoai, niranjan }@cs.stonybrook.edu Abstract Anticipating future actions in a video is use- ful for many autonomous and assistive tech- nologies. Most prior action anticipation work treat this as a vision modality problem, where the models learn the task information primar- ily from the video features in the action antic- ipation datasets. However, knowledge about action sequences can also be obtained from external textual data. In this work, we show how knowledge in pretrained language models can be adapted and distilled into vision-based action anticipation models. We show that a simple distillation technique can achieve effec- tive knowledge transfer and provide consistent gains on a strong vision model (Anticipative Vision Transformer) for two action anticipation datasets (3.5% relative gain on EGTEA-GAZE+ and 7.2% relative gain on EPIC-KITCHEN 55), giving a new state-of-the-art result1. 1 Introduction Anticipating future actions in the video of an un- folding scenario is an important capability for many applications in augmented reality (Salamin et al., 2006; Azuma, 2004), robotics (Duarte et al., 2018; Schydlo et al., 2018), and autonomous driv- ing (Chaabane et al., 2020; Suzuki et al., 2018). Anticipating what actions will likely happen in a scenario, requires one to both recognize what has happened so far, and use anticipative general knowl- edge about how action sequences tend to play out. Most models for this task use a pre-trained video encoder to extract information about what has hap- pened so far in the scenario, and use a text-based decoder to predict what action is likely to happen in the future (Carion et al., 2020; Dessalene et al., 2021; Liu et al., 2020; Sener et al., 2020). However, when trained on the target video datasets, the generalization of the models depends 1The models and code used are available at:https://github.com/StonyBrookNLP/action-anticipation- lmtovideo Figure 1: A model learning the action anticipation from only the vision modality (video frames) is essentially exposed to a very limited set of action sequences. Lan- guage models, which are pre-trained on large-scale text, can learn this distribution from the task, and a much larger domain-relevant text. We propose distilling this knowledge from text modality models to vision modal- ity model for video action anticipation task. on how well these video datasets cover the space of action sequence distributions. In other words, the knowledge that is learnt for predicting future ac- tions is, in effect, limited to the information in the target video datasets, where obtaining large scale coverage of action sequences is difficult. Knowledge about action sequences can also be obtained from text resources at scale. Lan- guage models, (e.g. BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019b)), are typically pre- trained on large collections of unlabeled texts with billions of tokens, where they acquire a wide- variety of …

Content truncated to 500 words