Harry Potter and the Action Prediction Challenge from Natural Language

Proceedings of NAACL-HLT 2019, pages 2124–2130 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2124 Harry Potter and the Action Prediction Challenge from Natural Language David Vilares Universidade da Coru˜na, CITIC Departamento de Computaci´on Campus de Elvi˜na s/n, 15071 A Coru˜na, Spain david.vilares@udc.es Carlos G´omez-Rodr´ıguez Universidade da Coru˜na, CITIC Departamento de Computaci´on Campus de Elvi˜na s/n, 15071 A Coru˜na, Spain carlos.gomez@udc.es Abstract We explore the challenge of action prediction from textual descriptions of scenes, a testbed to approximate whether text inference can be used to predict upcoming actions. As a case of study, we consider the world of the Harry Potter fantasy novels and inferring what spell will be cast next given a fragment of a story. Spells act as keywords that abstract actions (e.g. ‘Alohomora’ to open a door) and de- note a response to the environment. This idea is used to automatically build HPAC, a corpus containing 82 836 samples and 85 actions. We then evaluate different baselines. Among the tested models, an LSTM-based approach ob- tains the best performance for frequent actions and large scene descriptions, but approaches such as logistic regression behave well on in- frequent actions. 1 Introduction Natural language processing (NLP) has achieved significant advances in reading comprehension tasks (Chen et al., 2016; Salant and Berant, 2017). These are partially due to embedding methods (Mikolov et al., 2013; Devlin et al., 2018) and neural networks (Rosenblatt, 1958; Hochreiter and Schmidhuber, 1997; Vaswani et al., 2017), but also to the availability of new resources and challenges. For instance, in cloze-form tasks (Hermann et al., 2015; Bajgar et al., 2016), the goal is to predict the missing word given a short context. Weston et al. (2015) presented baBI, a set of proxy tasks for reading comprenhension. In the SQuAD corpus (Rajpurkar et al., 2016), the aim is to answer ques- tions given a Wikipedia passage. Kocisky et al. (2018) introduce NarrativeQA, where answering the questions requires to process entire stories. In a related line, Frermann et al. (2017) use fictional crime scene investigation data, from the CSI se- ries, to define a task where the models try to an- swer the question: ‘who committed the crime?’. In an alternative line of work, script induction (Schank and Abelson, 1977) has been also a use- ful approach to evaluate inference and semantic capabilities of NLP systems. Here, a model pro- cesses a document to infer new sequences that re- flect events that are statistically probable (e.g. go to a restaurant, be seated, check the menu, ...). For example, Chambers and Jurafsky (2008) in- troduce narrative event chains, a representation of structured knowledge of a set of events occur- ring around a protagonist. They then propose a method to learn statistical scripts, and also intro- duce two different evaluation strategies. With a related aim, Pichotta and Mooney (2014) propose a multi-event representation of statistical scripts to be able to consider multiple entities. These same authors (Pichotta and Mooney, 2016) have also studied the abilities …

Content truncated to 500 words