VSTAR: A Video-grounded Dialogue Dataset for Situated Semantic Understanding with Scene and Topic Transitions

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5036–5048 July 9-14, 2023 ©2023 Association for Computational Linguistics VSTAR: A Video-grounded Dialogue Dataset for Situated Semantic Understanding with Scene and Topic Transitions Yuxuan Wang1,2,3∗, Zilong Zheng2†, Xueliang Zhao1 Jinpeng Li1, Yueqian Wang1, Dongyan Zhao1,2,3,4† 1 Wangxuan Institute of Computer Technology, Peking University, Beijing, China 2 Beijing Institute for General Artificial Intelligence (BIGAI), Beijing, China 3 Center for Data Science, AAIS, Peking University, Beijing, China 4 National Key Laboratory of General Artificial Intelligence, Beijing, China {wyx,lijinpeng}@stu.pku.edu.cn, zlzheng@bigai.ai {xl.zhao,wangyueqian,zhaody}@pku.edu.cn https://vstar-benchmark.github.io/ 还有，你的新闻稿上有一些备 注。我只对前两页有意见。 Yeah, we're pretty lousy with pens around here, so knock yourself out. Really? Thanks. Here are a few notes on your press release. I only had a problem with the first two pages. It is two pages! Chicken korma from Rasika, and you're not gonna touch it? Sorry, I'm just not hungry. Scenei-1 Topicj-1 Topicj Topicj+1 Scenei 00:00 00:03 00:05 00:10 00:18 00:22 我们不喜欢太多笔, 你拿着 吧。 真的吗? 谢谢。 一共只有两页！ 拉西卡的腰果滑汁鸡，你都 不尝一下吗？ 抱歉，我只是不饿。 Figure 1: An overview of our VSTAR dataset. A 30 seconds video clip involves two dialogue scenes in which the environments and interlocutors are totally different. Abstract Video-grounded dialogue understanding is a challenging problem that requires machine to perceive, parse and reason over situated seman- tics extracted from weakly aligned video and dialogues. Most existing benchmarks treat both modalities the same as a frame-independent vi- sual understanding task, while neglecting the in- trinsic attributes in multimodal dialogues, such as scene and topic transitions. In this paper, we present Video-grounded Scene&Topic AwaRe dialogue (VSTAR) dataset, a large scale video-grounded dialogue understanding dataset based on 395 TV series. Based on VS- TAR, we propose two benchmarks for video- grounded dialogue understanding: scene seg- mentation and topic segmentation, and one benchmark for video-grounded dialogue gen- eration. Comprehensive experiments are per- formed on these benchmarks to demonstrate the importance of multimodal information and segments in video-grounded dialogue under- standing and generation. ∗This work was partially conducted when Yuxuan Wang was a research intern at BIGAI. †Correspondence to Zilong Zheng and Dongyan Zhao. 1 Introduction “Every film should have its own world, a logic and feel to it that expands beyond the exact image that the audience is see- ing.” — Christopher Nolan Have you seen movie “Memento”? If so, you may feel astonished for the mysterious movie plots but also clap for the excellent directing and edit- ing skills of director Christopher Nolan. In this movie, the main character plays a role of a murder victim with short-term memory. Truth, memories, fake stories are cut into pieces and re-ordered in the movie. Such nonlinear narrative structure is not unique to Nolan’s movie but is frequently used in story-based videos to increase the mysteriousness. The audience has to identify the type and chrono- logical order of each scene in order to understand the entire movie. We call such ability as scene transition identification. In our daily dialogue, similarly, topic shifting commonly occurs during chit-chatting. One …

Content truncated to 500 words