Video Question Answering with Phrases via Semantic Roles

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2460–2478 June 6–11, 2021. ©2021 Association for Computational Linguistics 2460 Video Question Answering with Phrases via Semantic Roles Arka Sadhu1 Kan Chen2 Ram Nevatia1 1University of Southern California 2Facebook Inc. {asadhu|nevatia}@usc.edu kanchen18@fb.com Abstract Video Question Answering (VidQA) evalua- tion metrics have been limited to a single-word answer or selecting a phrase from a fixed set of phrases. These metrics limit the VidQA models’ application scenario. In this work, we leverage semantic roles derived from video de- scriptions to mask out certain phrases, to in- troduce VidQAP which poses VidQA as a fill- in-the-phrase task. To enable evaluation of answer phrases, we compute the relative im- provement of the predicted answer compared to an empty string. To reduce the influence of language-bias in VidQA datasets, we re- trieve a video having a different answer for the same question. To facilitate research, we construct ActivityNet-SRL-QA and Charades- SRL-QA and benchmark them by extending three vision-language models. We perform ex- tensive analysis and ablative studies to guide future work. Code and data are public. 1 Introduction Given a video, Video Question Answering (VidQA) requires a model to provide an answer to a video related question. However, existing works treat VidQA as an N-way (N∼1k) classification task across a fixed set of phrases. Models trained un- der such formulations are strictly restricted in their recall rate, generalize poorly, and have severe limi- tations for end-user applications. In this work, we introduce Video Question Answering with Phrases (VidQAP) which treats VidQA as a fill-in-the-phrase task. Instead of a question, the input to VidQAP consists of a query expression with a query-token. Then, given a video, VidQAP requires replacing query-token with a se- quence of generated words. To generate a query, we leverage video descriptions and assign semantic roles to each phrase in these descriptions. Replac- ing a particular semantic-role with a query token produces a query-answer pair. We illustrate this in Figure 1 (details in Section 3.1). Video description: A man on top of a building throws a bowling ball towards the pins Q4: <Q-ARG0> throws a bowling ball towards the pins. Model’s generated answer: A man standing on a house Correct answer: A man on top of a building Q5: A man on top of a building <Q-V> a bowling ball towards the pins. Model’s generated answer: throws Correct answer: throws Q6: A man on top of a building throws <Q-ARG1> towards the pins. Model’s generated answer: a ball Correct answer: a bowling ball Q7: A man on top of a building throws a bowling ball <Q-ARG2> Model’s generated answer: towards some bottles Correct answer: towards the pins (b) Free-form Answer Generation ARG0 V ARG1 ARG2 Semantic Roles: Q1: Who throws a bowling ball towards the pins? Model’s Top Predictions: A: A man B: A man under the tree C: A person D: This boy Correct Answer: A man on top of a building …

Content truncated to 500 words