FIBER: Fill-in-the-Blanks as a Challenging Video Understanding Evaluation Framework

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2925 - 2940 May 22-27, 2022 c⃝2022 Association for Computational Linguistics FIBER: Fill-in-the-Blanks as a Challenging Video Understanding Evaluation Framework Santiago Castro Ruoyao Wang Pingxuan Huang Ian Stewart Oana Ignat Nan Liu Jonathan C. Stroud Rada Mihalcea University of Michigan – Ann Arbor, USA sacastro@umich.edu Two children throw _____ at each other as a video is captured in slow motion. _____ sits at a drum set and practices playing the drums. A boy is trying to comb his hair while _____ dries it. Correct answers: balloons, balloons filled with water, balloons of water, pink balloon, pink water balloon, things, wa- ter, water balloons, water-filled balloons Correct answers: child, drummer, fu- ture drummer, girl, kid, little girl, little kid, musician, small child, young girl Correct answers: another person, friend, girl, his sister, his sister with hairdryer, person, young woman Figure 1: Three examples from the FIBER dataset, each including three video frames, the caption, the blanked answers from the original caption together with the collected answers (all answers normalized, see Section 3.2). Abstract We propose fill-in-the-blanks as a video under- standing evaluation framework and introduce FIBER – a novel dataset consisting of 28,000 videos and descriptions in support of this evalu- ation framework. The fill-in-the-blanks setting tests a model’s understanding of a video by requiring it to predict a masked noun phrase in the caption of the video, given the video and the surrounding text. The FIBER bench- mark does not share the weaknesses of the cur- rent state-of-the-art language-informed video understanding tasks, namely: (1) video ques- tion answering using multiple-choice questions, where models perform relatively well because they exploit linguistic biases in the task for- mulation, thus making our framework chal- lenging for the current state-of-the-art systems to solve; and (2) video captioning, which re- lies on an open-ended evaluation framework that is often inaccurate because system answers may be perceived as incorrect if they differ in form from the ground truth. The FIBER dataset and our code are available at https: //lit.eecs.umich.edu/fiber/. 1 Introduction Despite current progress on multimodal (textual and visual) representations, language-informed video understanding is still a very challenging task for machine learning systems (Zhang et al., 2021; Li et al., 2021). This is due in large part to the task setup and the dataset construction. Current video understanding datasets often have at least one of two major limitations. First, they have limited application value. E.g., multiple-choice questions (Lei et al., 2018; Tapaswi et al., 2016; Jang et al., 2017; Castro et al., 2020) do not reflect real-world tasks. Second, they are based on sub- jective evaluation metrics, e.g., video captioning (Tran et al., 2016; Krishna et al., 2017; Zhou et al., 2018; Wang et al., 2019)), and are therefore hard to evaluate automatically, as the ground truth can be expressed in different ways. In this paper, we ad- dress these limitations by introducing a new dataset named FIBER that …

Content truncated to 500 words