Open-domain Video Commentary Generation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7326 - 7339 December 7-11, 2022 ©2022 Association for Computational Linguistics Open-Domain Video Commentary Generation Edison Marrese-Taylor1*, Yumi Hamazono1,2*, Tatsuya Ishigaki1, Goran Topic1 Yusuke Miyao1,3 Ichiro Kobayashi1,2, Hiroya Takamura1 National Institute of Advanced Industrial Science and Technology1 Ochanomizu University2, The University of Tokyo3 {edison.marrese,ishigaki.tatsuya}@aist.go.jp {goran.topic,takamura.hiroya}@aist.go.jp {hamazono.yumi,koba}@is.ocha.ac.jp, yusuke@is.s.u-tokyo.ac.jp *Authors contributed equally to this work. Abstract Live commentary plays an important role in sports broadcasts and video games, making spectators more excited and immersed. In this context, though approaches for automatically generating such commentary have been pro- posed in the past, they have been generally con- cerned with specific fields, where it is possible to leverage domain-specific information. In light of this, we propose the task of generating video commentary in an open-domain fashion. We detail the construction of a new large-scale dataset of transcribed commentary aligned with videos containing various human actions in a variety of domains, and propose approaches based on well-known neural architectures to tackle the task. To understand the strengths and limitations of current approaches, we present an in-depth empirical study based on our data. Our results suggest clear trade-offs between textual and visual inputs for the models and highlight the importance of relying on external knowledge in this open-domain setting, result- ing in a set of robust baselines for our task. 1 Introduction When watching TV programs or online live streams we find that the visual content is often accompanied by objective statements or subjective remarks about the events in the video given by a commentator in real time, aiming to help the spectators understand events in the videos. It has been shown that such live commentaries play an important role in sports broadcasts and video game matches or streams, for example, making spectators more excited, more immersed, and better informed about the content they are viewing (Schaffrath, 2003), thus enhanc- ing the value of online and home videos (Ishigaki et al., 2021). In view of this, the task of automati- cally generating such live commentaries on specific domains, such as sports (Kim and Choi, 2020) or video games (Ishigaki et al., 2021) has been pro- posed recently, with models often relying on field- specific information to aid the generation. Live commentary, in this paper, refers to commentary that can be listened to as audio or read as subtitle together with the original visual content. In light of this, in this paper we depart from previ- ous work on automatic live commentary generation by proposing an open-domain setting, where the goal is to enable models to generate live commen- taries for videos containing actions in a variety of situations. Compared with related work (Taniguchi et al., 2019; Kim and Choi, 2020) and in-domain versions of our task (Ishigaki et al., 2021), ours is particularly challenging as it keeps us from us- ing domain-specific features, which have proven essential for attaining good performance. Our task can be seen as a combination of funda- mental tasks …

Content truncated to 500 words