Event Ordering with a Generalized Model for Sieve Prediction Ranking

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 843–853, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP Event Ordering with a Generalized Model for Sieve Prediction Ranking Bill McDowell The Pennsylvania State University forkunited@gmail.com Nathanael Chambers United States Naval Academy nchamber@usna.edu Alexander G. Ororbia II The Pennsylvania State University ago109@psu.edu David Reitter The Pennsylvania State University reitter@psu.edu Abstract This paper improves on several aspects of a sieve-based event ordering architecture, CAEVO (Chambers et al., 2014), which creates globally consistent temporal rela- tions between events and time expressions. First, we examine the usage of word em- beddings and semantic role features. With the incorporation of these new features, we demonstrate a 5% relative F1 gain over our replicated version of CAEVO. Second, we reformulate the architecture’s sieve-based inference algorithm as a prediction rerank- ing method that approximately optimizes a scoring function computed using classifier precisions. Within this prediction rerank- ing framework, we propose an alternative scoring function, showing an 8.8% relative gain over the original CAEVO. We further include an in-depth analysis of one of the main datasets that is used to evaluate tem- poral classifiers, and we show that in spite of the density of this corpus, there is still a danger of overfitting. While this paper focuses on temporal ordering, its results are applicable to other areas that use sieve- based architectures. 1 Introduction Narratives that describe a series of events rarely do so in order. Basic rules of journalism dictate that important information leads a news report, and ac- cordingly, algorithms that re-order events chrono- logically need to combine a wealth of contextual, rhetorical, and commonsense information. Most research on event ordering aims to pro- duce only partial orderings of event mentions and time expressions (Bethard and Martin, 2007; Cheng et al., 2007; UzZaman and Allen, 2010; Llorens et al., 2010; Bethard, 2013). In the past, labeled corpora used for training and evaluation contained only small subsets of pairs of events and times. Some of these corpora, like Time- bank, have annotations restricted to salient, eas- ily labeled pairs within the same document. Other more recent data sets contain annotations that form timelines of events that involve common entities (Pustejovsky et al., 2003; Minard et al., 2015). Due to the lack of consistency of annotations across event pairs, it is difficult to use these cor- pora in accurately measuring the practical perfor- mance of event ordering algorithms. Richer datasets are becoming available that pro- vide more complete event orderings which in- clude logically implied relations that are less ev- ident from local text features. In particular, the TimeBank-Dense corpus provides a significantly more dense and complete set of annotations, al- lowing for the evaluation of methods that make use of broad contextual information across many event pairs (Cassidy et al., 2014). One method that has been developed to leverage such information is CAEVO—a sieve-based architecture that made the first effort toward dense event ordering (Cham- bers et al., 2014). This method …

Content truncated to 500 words