What is More Likely to Happen Next? Video-and-Language Future Event Prediction

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8769–8784, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8769 What is More Likely to Happen Next? Video-and-Language Future Event Prediction Jie Lei Licheng Yu Tamara L. Berg Department of Computer ScienceMohit Bansal University of North Carolina at Chapel Hill {jielei, licheng, tlberg, mbansal}@cs.unc.edu Abstract Given a video with aligned dialogue, peo- ple can often infer what is more likely to happen next. Making such predictions re- quires not only a deep understanding of the rich dynamics underlying the video and dia- logue, but also a significant amount of com- monsense knowledge. In this work, we ex- plore whether AI models are able to learn to make such multimodal commonsense next- event predictions. To support research in this direction, we collect a new dataset, named Video-and-Language Event Prediction (VLEP), with 28,726 future event predic- tion examples (along with their rationales) from 10,234 diverse TV Show and YouTube Lifestyle Vlog video clips. In order to pro- mote the collection of non-trivial challenging examples, we employ an adversarial human- and-model-in-the-loop data collection proce- dure. We also present a strong baseline in- corporating information from video, dialogue, and commonsense knowledge. Experiments show that each type of information is useful for this challenging task, and that compared to the high human performance on VLEP, our model provides a good starting point but leaves large room for future work.1 1 Introduction Given a video clip (premise event), humans can often describe logical events that might happen next (future events), and interestingly people tend to agree on which future events are more likely to happen than others. Making such predictions requires not only a deep understanding of the rich dynamics underlying the video and dialogue, but also a significant amount of multimodal common- sense knowledge about the world. In Figure 1 (top), 1Dataset, code are available at https://github. com/jayleicn/VideoLanguageFuturePred 00:21,320 --> 00:23,381 [Mark] Oh yeah! Maybe a shake. (Premise Summary : A woman with a white shirt with black buttons grinds fruit slush in a blender.) A. The woman in the white shirt pours the slush into a cup. Premise Event (Which event is more likely to happen right after the premise?) (Rationale: There are hollowed out watermelon rinds sitting around the blender.) (Rationale: Slushy drinks are more commonly served in a cup, but there are hollowed out watermelon rinds sitting around the blender.) B. The woman in the white shirt pours the slush into a watermelon rind and passes it to Mark. Future Events 00:26,436 --> 00:31,230 Dean: When I got back to my apartment, that phone was on my doormat. It had a text on it. (Premise Summary: The man being questioned references finding the cell phone in the evidence bag and there being a text on it. Detective Beckett reaches toward the evidence bag.) A. Beckett takes the phone and reads the text. Premise Event (Which event is more likely to happen right after the premise?) (Rationale: The detective probably …

Content truncated to 500 words