Partners in Crime: Multi-view Sequential Inference for Movie Understanding

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2057–2067, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2057 Partners in Crime: Multi-view Sequential Inference for Movie Understanding Nikos Papasarantopoulos♠ Lea Frermann♦ Mirella Lapata♠ Shay B. Cohen♠ ♠School of Informatics, University of Edinburgh, UK ♦School of Computing and Information Systems, University of Melbourne, Australia nikos.papasa@ed.ac.uk, lea.frermann@unimelb.edu.au mlap@inf.ed.ac.uk, scohen@inf.ed.ac.uk Abstract Multi-view learning algorithms are powerful representation learning tools, often exploited in the context of multimodal problems. How- ever, for problems requiring inference at the token-level of a sequence (that is, a separate prediction must be made for every time step), it is often the case that single-view systems are used, or that more than one views are fused in a simple manner. We describe an incremental neural architecture paired with a novel training objective for incremental inference. The net- work operates on multi-view data. We demon- strate the effectiveness of our approach on the problem of predicting perpetrators in crime drama series, for which our model signifi- cantly outperforms previous work and strong baselines. Moreover, we introduce two tasks, crime case and speaker type tagging, that con- tribute to movie understanding and demon- strate the effectiveness of our model on them.1 1 Introduction While many natural language processing (NLP) problems concern exclusively textual or speech data, the integration of multimodal information (such as images, video or audio) is beneficial for a variety of problems. For example, visual informa- tion has been used in affect analysis (Kahou et al., 2016), sentiment analysis (Morency et al., 2011) and machine translation (Calixto et al., 2017; Lala and Specia, 2018). This is also the case for prob- lems which are sequential in nature, such as video summarization (Smith and Kanade, 1998), contin- 1Our code is available at https://github.com/ papagandalf/multiview_csi. uous prediction of affect (Nicolaou et al., 2011) or engagement level prediction (Rehg et al., 2013). Most existing multi-view representation learn- ing approaches are tested in an unsupervised setup where the multi-view representations are learned separately from the task, and are designed to accommodate the learning of representation for monolithic (albeit multi-view) data points, not se- quences (see Wang et al. 2015 for a survey). In this paper, we propose a neural architecture cou- pled with a novel training objective that integrates multi-view information for sequence prediction problems. Our model creates a multimodal em- bedding for every element of a sequence by us- ing the correlational Gated Recurrent Unit (cor- rGRU; Yang et al. 2017) and makes token-level predictions based on those embeddings. Our train- ing objective combines a supervision-guided term (cross-entropy) with a multi-view correlation ob- jective on the available modalities. We demonstrate the effectiveness of our model in an incremental inference setup (Figure 1), where it makes predictions on the fly without en- coding the sequence in full, a more realistic sce- nario of interacting with data. This is a critical fea- ture for online applications such …

Content truncated to 500 words