Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14820–14835 July 9-14, 2023 ©2023 Association for Computational Linguistics Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models Qingyu Tan ∗ 1, 2 Hwee Tou Ng † 2 Lidong Bing1 1DAMO Academy, Alibaba Group 2Department of Computer Science, National University of Singapore {qingyu.tan,l.bing}@alibaba-inc.com {qtan6,nght}@comp.nus.edu.sg Abstract Reasoning about time is of fundamental impor- tance. Many facts are time-dependent. For ex- ample, athletes change teams from time to time, and different government officials are elected periodically. Previous time-dependent question answering (QA) datasets tend to be biased in either their coverage of time spans or question types. In this paper, we introduce a comprehen- sive probing dataset TEMPREASON to evaluate the temporal reasoning capability of large lan- guage models. Our dataset includes questions of three temporal reasoning levels. In addition, we also propose a novel learning framework to improve the temporal reasoning capability of large language models, based on temporal span extraction and time-sensitive reinforce- ment learning. We conducted experiments in closed book QA, open book QA, and reasoning QA settings and demonstrated the effectiveness of our approach1. 1 Introduction In recent years, large language models (LLMs) have achieved significant success in many natural language processing (NLP) tasks, such as natural language understanding (NLU) (Fei et al., 2023), information extraction (IE) (Ding et al., 2023), and question answering (QA) (Ye et al., 2023; Zhao et al., 2023). Many facts and answers are dependent on their related time scopes, such as ‘What soccer club was Lionel Messi playing for?’. Chia et al. (2022) has pointed out around 48% of the qualifiers in the widely-used knowledge base Wikidata (Vran- deˇci´c and Krötzsch, 2014) are time-related. That is, a significant number of the knowledge triples in the Wikidata KB have their expiry dates. Cor- rect understanding of temporal concepts is crucial ∗Qingyu Tan is under the Joint PhD Program between Alibaba and NUS. † Corresponding author. 1Our code and data are released on https://github.com/ DAMO-NLP-SG/TempReason for language models to be successful in real-world applications. To examine the temporal reasoning capabilities of LLMs, the Time-Sensitive Question Answering (TSQA) task has been proposed and sev- eral evaluation datasets were published for research purposes. The Time-sensitive QA dataset (Chen et al., 2021) and the TEMPLAMA dataset (Dhin- gra et al., 2022) were constructed based on the Wikidata temporal KB. StreamingQA (Liska et al., 2022) was constructed by news article collections in English WMT challenges from 2007 to 2020. One consensus of prior work is that time-sensitive QA is a challenging task and its performance is still far below human performance. However, they did not provide a systematic analysis of LM’s tempo- ral reasoning capability. In this paper, we aim to systematically analyze such capability and identify the strengths and weaknesses of LMs on temporal reasoning. As shown in Figure 1, humans’ understanding of temporal reasoning could be broken down into three levels: time-time (L1) relation, time-event (L2) relation, and …

Content truncated to 500 words