Script Induction as Language Modeling

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1681–1686, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Script Induction as Language Modeling Rachel Rudinger1, Pushpendre Rastogi1, Francis Ferraro1, and Benjamin Van Durme1,2 1Center for Language and Speech Processing 2Human Language Technology Center of Excellence Johns Hopkins University Abstract The narrative cloze is an evaluation met- ric commonly used for work on automatic script induction. While prior work in this area has focused on count-based meth- ods from distributional semantics, such as pointwise mutual information, we argue that the narrative cloze can be productively reframed as a language modeling task. By training a discriminative language model for this task, we attain improvements of up to 27 percent over prior methods on stan- dard narrative cloze metrics. 1 Introduction Although the concept of scripts in artificial intelli- gence dates back to the 1970s (Schank and Abel- son, 1977), interest in this topic has renewed with recent efforts to automatically induce scripts from text on a large scale. One particularly influential work in this area, Chambers and Jurafsky (2008), treats the problem of script induction as one of learning narrative chains, which they accomplish using simple textual co-occurrence statistics. For the novel task of learning narrative chains, they introduce a new evaluation metric, the narrative cloze test, which involves predicting a missing event from a chain of events drawn from text. Several follow-up works (Chambers and Jurafsky, 2009; Jans et al., 2012; Pichotta and Mooney, 2014; Rudinger et al., 2015) employ and ex- tend Chambers and Jurafsky (2008)’s methods for learning narrative chains, each using the narrative cloze to evaluate their work. 1 In this paper, we take the position that the nar- rative cloze test, which has been treated predom- 1A number of related works on script induction use alter- native task formulations and evaluations. (Chambers, 2013; Cheung et al., 2013; Cheung and Penn, 2013; Frermann et al., 2014; Manshadi et al., 2008; Modi and Titov, 2014; Reg- neri et al., 2010) inantly as a method for evaluating script knowl- edge, is more productively thought of simply as a language modeling task.2 To support this claim, we demonstrate a marked improvement over pre- vious methods on this task using a powerful dis- criminative language model – the Log-Bilinear model (LBL). Based on this finding, we believe one of the following conclusions must follow: ei- ther discriminative language models are a more effective technique for script induction than pre- vious methods, or the narrative cloze test is not a suitable evaluation for this task.3 2 Task Definition Following the definitions of Chambers and Juraf- sky (2008), a narrative chain is “a partially or- dered set of narrative events that share a common actor,” where a narrative event is “a tuple of an event (most simply a verb) and its participants, represented as typed dependencies.” (De Marneffe et al., 2006) Formally, e := (v, d), where e is a narrative event, v is a verb lemma, and …

Content truncated to 500 words