Movie101: A New Movie Understanding Benchmark

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4669–4684 July 9-14, 2023 ©2023 Association for Computational Linguistics Movie101: A New Movie Understanding Benchmark Zihao Yue, Qi Zhang, Anwen Hu, Liang Zhang, Ziheng Wang, Qin Jin∗ School of Information, Renmin University of China {yzihao, zhangqi1996, anwenhu, zhangliang00, zihengwang, qjin}@ruc.edu.cn Abstract To help the visually impaired enjoy movies, au- tomatic movie narrating systems are expected to narrate accurate, coherent, and role-aware plots when there are no speaking lines of ac- tors. Existing works benchmark this challenge as a normal video captioning task via some simplifications, such as removing role names and evaluating narrations with ngram-based metrics, which makes it difficult for automatic systems to meet the needs of real application scenarios. To narrow this gap, we construct a large-scale Chinese movie benchmark, named Movie101. Closer to real scenarios, the Movie Clip Narrating (MCN) task in our benchmark asks models to generate role-aware narration paragraphs for complete movie clips where no actors are speaking. External knowledge, such as role information and movie genres, is also provided for better movie understanding. Be- sides, we propose a new metric called Movie Narration Score (MNScore) for movie narrat- ing evaluation, which achieves the best corre- lation with human evaluation. Our benchmark also supports the Temporal Narration Ground- ing (TNG) task to investigate clip localization given text descriptions. For both two tasks, our proposed methods well leverage external knowledge and outperform carefully designed baselines. The dataset and codes are released at https://github.com/yuezih/Movie101. 1 Introduction The estimated number of visually impaired people worldwide was about 285 million by 2020, accord- ing to reports (He et al., 2020). While regulations are in place to ensure increased access for these audiences to experience the culturally dominant movies and TV shows on popular media platforms, technologies that provide them with genuine expe- rience are becoming increasingly important. Audio description (AD, also known as video description) ∗*Corresponding Author. is a form of such technology intended for visu- ally impaired audiences to experience the movie or TV show by hearing what is happening on-screen. However, producing movie narration scripts is not trivial, often requiring a professional writer to over- see the original movie. The high cost of narration generation (Lakritz and Salway, 2006) greatly hin- ders the production of movies with AD and thus limits the opportunities for visually impaired users to experience movies. To address this issue, attempts have been car- ried out to automate AD production. Datasets of movies with ADs are constructed to support the research on automatic AD generation, including the MPII-MD dataset (Rohrbach et al., 2015) and M-VAD dataset (Torabi et al., 2015), with shot- level ADs or scripts aligned to the visual contents of movie. Consequently, different solutions for au- tomatic movie narrating have been proposed based on these datasets (Rohrbach et al., 2017). However, existing benchmarks suffer from sev- eral limitations. Firstly, there is a gap between the designed tasks and the actual movie narration scenario. These …

Content truncated to 500 words