Neural Models of Factuality

Proceedings of NAACL-HLT 2018, pages 731–744 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Neural models of factuality Rachel Rudinger Johns Hopkins University Aaron Steven White University of Rochester Benjamin Van Durme Johns Hopkins University Abstract We present two neural models for event fac- tuality prediction, which yield significant per- formance gains over previous models on three event factuality datasets: FactBank, UW, and MEANTIME. We also present a substantial expansion of the It Happened portion of the Universal Decompositional Semantics dataset, yielding the largest event factuality dataset to date. We report model results on this extended factuality dataset as well. 1 Introduction A central function of natural language is to convey information about the properties of events. Per- haps the most fundamental of these properties is factuality: whether an event happened or not. A natural language understanding system’s abil- ity to accurately predict event factuality is impor- tant for supporting downstream inferences that are based on those events. For instance, if we aim to construct a knowledge base of events and their par- ticipants, it is crucial that we know which events to include and which ones not to. The event factuality prediction task (EFP) in- volves labeling event-denoting phrases (or their heads) with the (non)factuality of the events de- noted by those phrases (Saur´ı and Pustejovsky, 2009, 2012; de Marneffe et al., 2012). Figure 1 exemplifies such an annotation for the phrase headed by leave in (1), which denotes a factual event (⊕=factual, ⊖=nonfactual). (1) Jo failed to leave no trace. ⊕ In this paper, we present two neural models of event factuality (and several variants thereof). We show that these models significantly outperform previous systems on four existing event factual- ity datasets – FactBank (Saur´ı and Pustejovsky, 2009), the UW dataset (Lee et al., 2015), MEAN- TIME (Minard et al., 2016), and Universal De- failed Jo leave to trace no . + Inside context Outside context Figure 1: Event factuality (⊕=factual) and inside v. outside context for leave in the dependency tree. compositional Semantics It Happened v1 (UDS- IH1; White et al., 2016) – and we demonstrate the efficacy of multi-task training and ensembling in this setting. In addition, we collect and release an extension of the UDS-IH1 dataset, which we refer to as UDS-IH2, to cover the entirety of the English Universal Dependencies v1.2 (EUD1.2) treebank (Nivre et al., 2015), thereby yielding the largest event factuality dataset to date.1 We begin with theoretical motivation for the models we propose as well as discussion of prior EFP datasets and systems (§2). We then describe our own extension of the UDS-IH1 dataset (§3), followed by our neural models (§4). Using the data we collect, along with the existing datasets, we evaluate our models (§6) in five experimental settings (§5) and analyze the results (§7). 2 Background 2.1 Linguistic description Words from effectively every syntactic category can convey information about the factuality of an event. For instance, negation (2a), modal auxil- iaries (2b), determiners (2c), …

Content truncated to 500 words