Using Sentence-Level LSTM Language Models for Script Inference

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 279–289, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Using Sentence-Level LSTM Language Models for Script Inference Karl Pichotta Department of Computer Science The University of Texas at Austin pichotta@cs.utexas.edu Raymond J. Mooney Department of Computer Science The University of Texas at Austin mooney@cs.utexas.edu Abstract There is a small but growing body of research on statistical scripts, models of event sequences that allow probabilistic inference of implicit events from docu- ments. These systems operate on struc- tured verb-argument events produced by an NLP pipeline. We compare these sys- tems with recent Recurrent Neural Net models that directly operate on raw tokens to predict sentences, finding the latter to be roughly comparable to the former in terms of predicting missing events in documents. 1 Introduction Statistical scripts are probabilistic models of event sequences (Chambers and Jurafsky, 2008). learned script model is capable of processing aA document and inferring events that are probable but not explicitly stated. These models operate on automatically extracted structured events (for ex- ample, verbs with entity arguments), which are de- rived from standard NLP tools such as dependency parsers and coreference resolution engines. Recent work has demonstrated that standard se- quence models applied to such extracted event sequences, e.g. discriminative language models (Rudinger et al., 2015) and Long Short Term Memory (LSTM) recurrent neural nets (Pichotta and Mooney, 2016), are able to infer held-out events more accurately than previous approaches. These results call into question the extent to which statistical event inference systems require linguis- tic preprocessing and syntactic structure. In an at- tempt to shed light on this issue, we compare ex- isting script models to LSTMs trained as sentence- level language models which try to predict the sequence of words in the next sentence from a learned representation of the previous sentences using no linguistic preprocessing. Some prior statistical script learning systems are focused on knowledge induction. These sys- tems are primarily designed to induce collections of co-occurring event types involving the same en- tities, and their ability to infer held-out events is not their primary intended purpose (Chambers and Jurafsky, 2008; Ferraro and Van Durme, 2016, in- ter alia). In the present work, we instead investi- gate the behavior of systems trained to directly op- timize performance on the task of predicting sub- sequent events; in other words, we are investigat- ing statistical models of events in discourse. Much prior research on statistical script learn- ing has also evaluated on inferring missing events from documents. However, the exact form that this task takes depends on the adopted definition of what constitutes an event: in previous work, events are defined in different ways, with differ- ing degrees of structure. We consider simply us- ing raw text, which requires no explicit syntactic annotation, as our mediating representation, and evaluate how raw text models compare to models of more structured events. Kiros et al. (2015) introduced skip-thought vec- tor models, in which …

Content truncated to 500 words