Whodunnit? Crime Drama as a Case for Natural Language Understanding

Whodunnit? Crime Drama as a Case for Natural Language Understanding Lea Frermann Shay B. Cohen Mirella Lapata Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB l.frermann@ed.ac.uk scohen@inf.ed.ac.uk mlap@inf.ed.ac.uk Abstract In this paper we argue that crime drama ex- emplified in television programs such as CSI: Crime Scene Investigation is an ideal testbed for approximating real-world natural language understanding and the complex inferences as- sociated with it. We propose to treat crime drama as a new inference task, capitalizing on the fact that each episode poses the same ba- sic question (i.e., who committed the crime) and naturally provides the answer when the perpetrator is revealed. We develop a new dataset1 based on CSI episodes, formalize per- petrator identification as a sequence labeling problem, and develop an LSTM-based model which learns from multi-modal data. Exper- imental results show that an incremental in- ference strategy is key to making accurate guesses as well as learning from representa- tions fusing textual, visual, and acoustic input. 1 Introduction The success of neural networks in a variety of ap- plications (Sutskever et al., 2014; Vinyals et al., 2015) and the creation of large-scale datasets have played a critical role in advancing machine under- standing of natural language on its own or together with other modalities. The problem has assumed several guises in the literature such as reading com- prehension (Richardson et al., 2013; Rajpurkar et al., 2016), recognizing textual entailment (Bowman et al., 2015; Rockt¨aschel et al., 2016), and notably question answering based on text (Hermann et al., 1Our dataset is available at https://github.com/ EdinburghNLP/csi-corpus. 2015; Weston et al., 2015), images (Antol et al., 2015), or video (Tapaswi et al., 2016). In order to make the problem tractable and amenable to computational modeling, existing ap- proaches study isolated aspects of natural language understanding. For example, it is assumed that un- derstanding is an offline process, models are ex- pected to digest large amounts of data before being able to answer a question, or make inferences. They are typically exposed to non-conversational texts or still images when focusing on the visual modality, ignoring the fact that understanding is situated in time and space and involves interactions between speakers. In this work we relax some of these sim- plifications by advocating a new task for natural lan- guage understanding which is multi-modal, exhibits spoken conversation, and is incremental, i.e., un- folds sequentially in time. Specifically, we argue that crime drama exempli- fied in television programs such as CSI: Crime Scene Investigation can be used to approximate real-world natural language understanding and the complex in- ferences associated with it. CSI revolves around a team of forensic investigators trained to solve crim- inal cases by scouring the crime scene, collecting irrefutable evidence, and finding the missing pieces that solve the mystery. Each episode poses the same “whodunnit” question and naturally provides the an- swer when the perpetrator is revealed. Speculation about the identity of the perpetrator is an integral …

Content truncated to 500 words