Reasoning with Language Model Prompting: A Survey

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 487–507 July 9-14, 2023 ©2023 Association for Computational Linguistics Revealing Single Frame Bias for Video-and-Language Learning Jie Lei Tamara L. Berg Mohit Bansal Department of Computer Science University of North Carolina at Chapel Hill {jielei, tlberg, mbansal}@cs.unc.edu Abstract Training an effective video-and-language model intuitively requires multiple frames as model inputs. However, it is unclear whether using multiple frames is beneficial to down- stream tasks, and if yes, whether the perfor- mance gain is worth the drastically-increased computation and memory costs resulting from using more frames. In this work, we explore single-frame models for video-and-language learning. On a diverse set of video-and- language tasks (including text-to-video re- trieval and video question answering), we show the surprising result that, with large-scale pre- training and a proper frame ensemble strategy at inference time, a single-frame trained model that does not consider temporal information can achieve better performance than existing methods that use multiple frames for training. This result reveals the existence of a strong “static appearance bias” in popular video-and- language datasets. Therefore, to allow for a more comprehensive evaluation of video- and-language models, we propose two new re- trieval tasks based on existing fine-grained ac- tion recognition datasets that encourage tempo- ral modeling. Our code is available at https: //github.com/jayleicn/singularity. 1 Introduction Video and language are the two primary signals that constitute much of the world we perceive ev- ery day – we observe our surrounding environment with our eyes in the form of continuous visual input (video), and communicate with others via language. Intuitively, this leads one to assume that training an effective video-and-language model should re- quire multiple video frames as input. Standard methods (Zhu and Yang, 2020; Xu et al., 2021; Li et al., 2020a; Luo et al., 2021) in this area typi- cally use multiple densely sampled frames for train- ing. Recent work (Lei et al., 2021) proposes sparse sampling for video-and-language understanding, where it claims that a few sparsely sampled clips are sufficient for learning due to the high redun- dancy in videos. This technique has shown (Lei et al., 2021; Zellers et al., 2021) to be successful in various video-language benchmarks (Jang et al., 2017; Xu et al., 2016; Anne Hendricks et al., 2017; Krishna et al., 2017a; Xu et al., 2017; Yu et al., 2018; Lei et al., 2018). However, as demonstrated by Bain et al. (2021); Luo et al. (2021); Lei et al. (2021), training with fewer frames (e.g., a single frame) leads to significantly worse performance compared to their multi-frame counterparts. In con- trast, in this work, we show that with proper model- ing, single-frame models could achieve competitive performance, hence revealing “static appearance bias” in popular video-and-language datasets. We start by building a standard image-language model, with a vision encoder and a language en- coder for image and text encoding, followed by a multi-modal encoder with cross-attention for cross- modal fusion. We pre-train the model …

Content truncated to 500 words