Temporal Reasoning in Natural Language Inference

Temporal Reasoning in Natural Language Inference Siddharth Vashishtha University of Rochester svashis3@cs.rochester.edu Adam Poliak Barnard College apoliak@barnard.edu Yash Kumar Lal Stony Brook University yash.lal@stonybrook.edu Benjamin Van Durme Johns Hopkins University vandurme@cs.jhu.edu Aaron Steven White University of Rochester aaron.white@rochester.edu Abstract We introduce five new natural language in- ference (NLI) datasets focused on temporal reasoning. We recast four existing datasets annotated for event duration—how long an event lasts—and event ordering—how events are temporally arranged—into more than one million NLI examples. We use these datasets to investigate how well neural models trained on a popular NLI corpus capture these forms of temporal reasoning. 1 Introduction The ability to reason about how events unfold in time is core to how humans structure their knowl- edge about the world (Casati and Varzi, 1996; Zacks and Tversky, 2001; Radvansky and Zacks, 2014), and modeling such temporal reasoning has been central to many classical AI approaches (Mc- Carthy and Hayes, 1987; Kahn and Gorry, 1977; McDermott, 1982; Allen, 1984; Kowalski and Ser- got, 1989; Pani and Bhattacharjee, 2001). Natural language supports various forms of tem- poral reasoning, including reasoning about the chronology and duration of events, and many Nat- ural Language Understanding (NLU) tasks and models have been employed for understanding and capturing different aspects of temporal rea- soning (UzZaman et al., 2013; Llorens et al., 2015; Mostafazadeh et al., 2016; Reimers et al., 2016; Tourille et al., 2017; Ning et al., 2017, 2018a; Meng and Rumshisky, 2018; Ning et al., 2018b; Han et al., 2019; Naik et al., 2019; Vashishtha et al., 2019; Zhou et al., 2019, 2020). More broadly, the ability to perform temporal reasoning is important for un- derstanding narratives (Nakhimovsky, 1987; Jung et al., 2011; Cheng et al., 2013), answering ques- tions (Bruce, 1972; Khashabi, 2019; Ning et al., 2020), and summarizing events (Jung et al., 2011; Wang et al., 2018). Order ▶ We waited until 2:25 PM and then left. The waiting started before the leaving started. ▶ Reggie said he will pay us soon. The paying ended before the saying started. Duration ▶ The greeter said there was about 15 mins waiting. The saying did take or will take shorter than an hour. ▶ Randy , this is the issue I left you the voice mail on. The leaving did take or will take longer than a day. Table 1: NLI sentence pairs from our recasted datasets. ▶ indicates the line is a context, and the following line is its corresponding hypothesis. Hypotheses in green indicate that the context entails the hypothesis; those in red indicate that it does not entail the hypothesis. Given that temporal reasoning is integral to natu- ral language understanding (NLU) and that Natural Language Inference (NLI) is a common framework for evaluating how well models capture semantic phenomena integral to NLU (Cooper et al., 1996; Dagan et al., 2006; White et al., 2017; Poliak et al., 2018), it is important to evaluate how well different classes of NLI models trained on common generic NLI datasets capture …

Content truncated to 500 words