Speaker Naming in Movies

Proceedings of NAACL-HLT 2018, pages 2206–2216 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Speaker Naming in Movies Mahmoud Azab, Mingzhe Wang, Max Smith, Noriyuki Kojima, Jia Deng, Rada Mihalcea Computer Science and Engineering, University of Michigan {mazab,mzwang,mxsmith,kojimano,jiadeng,mihalcea}@umich.edu Abstract We propose a new model for speaker naming in movies that leverages visual, textual, and acoustic modalities in an unified optimization framework. To evaluate the performance of our model, we introduce a new dataset con- sisting of six episodes of the Big Bang The- ory TV show and eighteen full movies cover- ing different genres. Our experiments show that our multimodal model significantly out- performs several competitive baselines on the average weighted F-score metric. To demon- strate the effectiveness of our framework, we design an end-to-end memory network model that leverages our speaker naming model and achieves state-of-the-art results on the subtitles task of the MovieQA 2017 Challenge. 1 Introduction Identifying speakers and their names in movies, and videos in general, is a primary task for many video analysis problems, including automatic sub- title labeling (Hu et al., 2015), content-based video indexing and retrieval (Zhang et al., 2009), video summarization (Tapaswi et al., 2014), and video storyline understanding (Tapaswi et al., 2014). It is a very challenging task, as the visual appearance of the characters changes over the course of the movie due to several factors such as scale, cloth- ing, illumination, and so forth (Arandjelovic and Zisserman, 2005; Everingham et al., 2006). The annotation of movie data with speakers’ names can be helpful in a number of applications, such as movie question answering (Tapaswi et al., 2016), automatic identification of character relationships (Zhang et al., 2009), or automatic movie caption- ing (Hu et al., 2015). Most previous studies relied primarily on vi- sual information (Arandjelovic and Zisserman, 2005; Everingham et al., 2006), and aimed for the slightly different task of face track labeling; 01:02:00%-->%01:02:01 Jack,%must you go? 01:02:01%-->%01:02:04 Time%for%me%to%go%row with other slaves. 01:02:07%-->%01:02:08 Good%night,%Rose. Names: Jack Rose …… Jack,%must% you%go? …… Good%night Jack:%Hold%On! Rose:%I%trust%you. Jack:%All%right!% Open%your%eyes. Input Inference Rose:I'm flying.%Jack. Output Joint%Optimization Figure 1: Overview of our approach for speaker naming. speakers who did not appear in the video frame were not assigned any names, which is common in movies and TV shows. Other available sources of information such as scripts were only used to ex- tract cues about the speakers’ names to associate the faces in the videos with their corresponding character name (Everingham et al., 2006; Tapaswi et al., 2015; B¨auml et al., 2013; Sivic et al., 2009); however since scripts are not always available, the applicability of these methods is somehow limited. Other studies focused on the problem of speaker recognition without naming, using the speech modality as a single source of information. While some of these studies attempted to incorporate the visual modality, their goal was to cluster the speech segments rather than name the speakers 2206 (Erzin et al., 2005; Bost and Linares, 2014; Kap- souras et al., 2015; Bredin …

Content truncated to 500 words