“Going on a vacation” takes longer than “Going for a walk”: A Study of Temporal Commonsense Understanding

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3363–3369, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3363 “Going on a vacation” takes longer than “Going for a walk”: A Study of Temporal Commonsense Understanding Ben Zhou,1 Daniel Khashabi,2 Qiang Ning,3 Dan Roth1 1University of Pennsylvania, 2Allen Institute for AI, 3University of Illinois at Urbana-Champaign {xyzhou,danroth}@cis.upenn.edu danielk@allenai.org qning2@illinois.edu Abstract Understanding time is crucial for understand- ing events expressed in natural language. Be- cause people rarely say the obvious, it is of- ten necessary to have commonsense knowl- edge about various temporal aspects of events, such as duration, frequency, and temporal or- der. However, this important problem has so far received limited attention. This pa- per systematically studies this temporal com- monsense problem. Specifically, we define five classes of temporal commonsense, and use crowdsourcing to develop a new dataset, MCTACO , that serves as a test set for this task. We find that the best current methods used on MCTACO are still far behind human performance, by about 20%, and discuss sev- eral directions for improvement. We hope that the new dataset and our study here can foster more future research on this topic.1 1 Introduction Natural language understanding requires the abil- ity to reason with commonsense knowledge (Schu- bert, 2002; Davis, 2014), and the last few years have seen significant amount of work in this direc- tion (e.g., Zhang et al. (2017); Bauer et al. (2018); Tandon et al. (2018)). This work studies a spe- cific type of commonsense: temporal common- sense. For instance, given two events “going on a vacation” and “going for a walk,” most humans would know that a vacation is typically longer and occurs less often than a walk, but it is still chal- lenging for computers to understand and reason about temporal commonsense. 1 The dataset, annotation interfaces, guidelines, and qual- ification tests are available at: https://cogcomp. seas.upenn.edu/page/publication_view/882. * This work was done while the second author was affil- iated with the University of Pennsylvania. Figure 1: Five types of temporal commonsense in MCTACO. Note that a question may have multiple correct answers. Temporal commonsense has received limited at- tention so far. Our first contribution is that, to the best of our knowledge, we are the first to systematically study and quantify performance on a range of temporal commonsense phenom- ena. Specifically, we consider five temporal prop- erties: duration (how long an event takes), tem- poral ordering (typical order of events), typical time (when an event happens), frequency (how of- ten an event occurs), and stationarity (whether a state holds for a very long time or indefinitely). Previous work has investigated some of these as- pects, either explicitly or implicitly (e.g., dura- tion (Gusev et al., 2011; Williams, 2012) and or- dering (Chklovski and Pantel, 2004; Ning et al., 2018b)), but none of them have defined or studied 3364 all aspects of temporal commonsense in a unified …

Content truncated to 500 words