Video Highlight Prediction Using Audience Chat Reactions

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 972–978 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Video Highlight Prediction Using Audience Chat Reactions Cheng-Yang Fu, Joon Lee, Mohit Bansal, Alex C. Berg UNC Chapel Hill {cyfu, joonlee, mbansal, aberg}@cs.unc.edu Abstract Sports channel video portals offer an ex- citing domain for research on multimodal, multilingual analysis. We present meth- ods addressing the problem of automatic video highlight prediction based on joint visual features and textual analysis of the real-world audience discourse with com- plex slang, in both English and tradi- tional Chinese. We present a novel dataset based on League of Legends champi- onships recorded from North American and Taiwanese Twitch.tv channels (will be released for further research), and demon- strate strong results on these using multi- modal, character-level CNN-RNN model architectures. 1 Introduction On-line eSports events provide a new setting for observing large-scale social interaction focused on a visual story that evolves over time—a video game. While watching sporting competitions has been a major source of entertainment for millen- nia, and is a significant part of today’s culture, eS- ports brings this to a new level on several fronts. One is the global reach, the same games are played around the world and across cultures by speak- ers of several languages. Another is the scale of on-line text-based discourse during matches that is public and amendable to analysis. One of the most popular games, League of Legends, drew 43 mil- lion views for the 2016 world series final matches (broadcast in 18 languages) and a peak concurrent viewership of 14.7 million1. Finally, players in- teract through what they see on screen while fans (and researchers) can see exactly the same views. 1http://www.lolesports.com/en_US/articles/ 2016-league-legends-world-championship-numbers (a) Twitch (b) Youtube (c) Facebook Figure 1: Pictures of Broadcasting platforms:(a) Twitch: League of Legends Tournament Broadcasting, (b) Youtube: News Channel, (c)Facebook: Personal live sharing This paper builds on the wealth of interaction around eSports to develop predictive models for match video highlights based on the audience’s online chat discourse as well as the visual record- ings of matches themselves. ESports journal- ists and fans create highlight videos of impor- tant moments in matches. Using these as ground truth, we explore automatic prediction of high- lights via multimodal CNN+RNN models for mul- tiple languages. Appealingly this task is natural, as the community already produces the ground truth and is global, allowing multilingual multi- modal grounding. Highlight prediction is about capturing the ex- citing moments in a specific video (a game match in this case), and depends on the context, the state of play, and the players. This task of predicting the exciting moments is hence different from sum- marizing the entire match into a story summary. Hence, highlight prediction can benefit from the available real-time text commentary from fans, which is valuable in exposing more abstract back- ground context, that may not be accessible with 972 computer vision techniques that can easily identify some aspects of the state …

Content truncated to 500 words