DeepMaven: Deep Question Answering on Long-Distance Movie/TV Show Videos with Multimedia Knowledge Extraction and Synthesis

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3041–3051 May 2-6, 2023 ©2023 Association for Computational Linguistics DeepMaven: Deep Question Answering on Long-Distance Movie/TV Show Videos with Multimedia Knowledge Extraction and Synthesis Yi Fung1∗, Han Wang2, Tong Wang2, Ali Kebarighotbi2, Mohit Bansal2, Heng Ji2, Prem Natarajan2 1University of Illinois at Urbana Champaign, 2Amazon Alexa yifung2@illinois.edu {wnghn,tonwng,alikeba,mobansal,jihj,premknat}@amazon.com Abstract Long video content understanding poses a challenging set of research questions as it in- volves long-distance, cross-media reasoning and knowledge awareness. In this paper, we present a new benchmark for this problem do- main, targeting the task of deep movie/TV ques- tion answering (QA) beyond previous work’s focus on simple plot summary and short video moment settings. We define several baselines based on direct retrieval of relevant context for long-distance movie QA. Observing that real- world QAs may require higher-order multi-hop inferences, we further propose a novel frame- work, called the DEEPMAVEN, which extracts events, entities, and relations from the rich mul- timedia content in long videos to preconstruct movie knowledge graphs (movieKGs), and at the time of QA inference, complements general semantics with structured knowledge for more effective information retrieval and knowledge reasoning. We also introduce our recently col- lected DeepMovieQA dataset, including 1,000 long-form QA pairs from 41 hours of videos, to serve as a new and useful resource for fu- ture work. Empirical results show the Deep- Maven performs competitively for both the new DeepMovieQA and the pre-existing MovieQA dataset.1 1 Introduction Our world tells an evolving story of people, ob- jects, and their interactions. This storytelling may exist in various forms, from textual summaries and spoken dialogues, to accompanying images and videos. Because of its dynamically evolving and multi-media nature, long-distance video question answering on movies/TV shows provides a useful setting for studying the computational understand- ing of interconnected stories and events that aligns closely with real-world application scenarios. Yet, ∗Work done as an intern at Amazon Alexa AI. 1See https://www.amazon.science/publications for update of information about code and resources. current intelligent systems still struggle with ad- equately processing the rich multimedia content in long videos and fail to answer many common inquiries that humans are interested in. If we ask a virtual assistant about the relationship between two main characters in a well-known movie/TV show, it likely defaults to some null response such as ªhmm... I don’t know this one". The research challenge of long video content understanding and question answering frameworks stems from the need to support information prob- ing on certain specific details over a large multime- dia context space. Propagating information across long-distance has been a well-known challenge due to the vanishing gradient and memory loss prob- lem (Hochreiter, 1998). Long videos also can not fit its entire data in-memory typically for end-to- end feature extraction and neural network training. Meanwhile, retrieval-based methodologies, which first find sections in the video relevant to a query through semantic cues from the corresponding dia- logues and/or frames, and then return the …

Content truncated to 500 words