Neural Language Modeling for Contextualized Temporal Graph Generation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 864–881 June 6–11, 2021. ©2021 Association for Computational Linguistics 864 Neural Language Modeling for Contextualized Temporal Graph Generation Aman Madaan, Yiming Yang Language Technologies Institute, Carnegie Mellon University Pittsburgh, PA, USA amadaan@cs.cmu.edu Abstract This paper presents the first study on using large-scale pre-trained language models for automated generation of an event-level tem- poral graph for a document. Despite the huge success of neural pre-training methods in NLP tasks, its potential for temporal reasoning over event graphs has not been sufficiently ex- plored. Part of the reason is the difficulty in obtaining large training corpora with human- annotated events and temporal links. We ad- dress this challenge by using existing IE/NLP tools to automatically generate a large quantity (89,000) of system-produced document-graph pairs, and propose a novel formulation of the contextualized graph generation problem as a sequence-to-sequence mapping task. These strategies enable us to leverage and fine-tune pre-trained language models on the system- induced training data for the graph generation task. Our experiments show that our approach is highly effective in generating structurally and semantically valid graphs. Further, eval- uation on a challenging hand-labeled, out-of- domain corpus shows that our method outper- forms the closest existing method by a large margin on several metrics. We also show a downstream application of our approach by adapting it to answer open-ended temporal questions in a reading comprehension setting.1 1 Introduction Temporal reasoning is crucial for analyzing the in- teractions among complex events and producing coherent interpretations of text data (Duran et al., 2007). There is a rich body of research on the use of temporal information in a variety of impor- tant application domains, including topic detection and tracking (Makkonen et al., 2003), information extraction (Ling and Weld, 2010), parsing of clin- ical records (Lin et al., 2016), discourse analy- 1Code and pre-trained models available at https:// github.com/madaan/temporal-graph-gen sis (Evers-Vermeul et al., 2017), and question an- swering (Ning et al., 2020). Graphs are a natural choice for representing the temporal ordering among events, where the nodes are the individual events, and the edges capture temporal relationships such as “before”, “after” or “simultaneous”. Representative work on automated extraction of such graphs from textual documents includes the early work by Chambers and Jurafsky (2009), where the focus is on the construction of event chains from a collection of documents, and the more recent CAEVO (Chambers et al., 2014) and Cogcomptime (Ning et al., 2018), which extract a graph for each input document instead. These methods focus on rule-based and statistical sub- modules to extract verb-centered events and the temporal relations among them. As an emerging area of NLP, large scale pre-trained language mod- els have made strides in addressing challenging tasks like commonsense knowledge graph comple- tion (Bosselut et al., 2019) and task-oriented dialog generation (Budzianowski and Vuli´c, 2019). These systems typically fine-tune large language models on a corpus of a task-specific dataset. However, these …

Content truncated to 500 words