CALM-Bench: A Multi-task Benchmark for Evaluating Causality-Aware Language Models

Findings of the Association for Computational Linguistics: EACL 2023, pages 296–311 May 2-6, 2023 ©2023 Association for Computational Linguistics CALM-Bench: A Multi-task Benchmark for Evaluating Causality Aware Language Models Dhairya Dalal † and Mihael Arcan‡ and Paul Buitelaar† ‡ †SFI Centre for Research and Training in Artificial Intelligence † ‡Insight SFI Research Centre for Data Analytics Data Science Institute, University of Galway d.dalal1@nuigalway.ie, {mihael.arcan,paul.buitelaar}@universityofgalway.ie Abstract Causal reasoning is a critical component of hu- man cognition and is required across a range of question-answering (QA) tasks (such as ab- ductive reasoning, commonsense QA, and pro- cedural reasoning). Research on causal QA has been underdefined, task-specific, and lim- ited in complexity. Recent advances in founda- tion language models (such as BERT, ERNIE, and T5) have shown the efficacy of pre-trained models across diverse QA tasks. However, there is limited research exploring the causal reasoning capabilities of those language mod- els and no standard evaluation benchmark. To unify causal QA research, we propose CALM- Bench, a multi-task benchmark for evaluating causality-aware language models (CALM). We present a standardized definition of causal QA tasks and show empirically that causal reason- ing can be generalized and transferred across different QA tasks. Additionally, we share a strong multi-task baseline model which out- performs single-task fine-tuned models on the CALM-Bench tasks. 1 Introduction Causal reasoning is a crucial aspect of human cognition and is critical to the development of our mental models of reality (Neeleman et al., 2012; Johnson-Laird and Khemlani, 2017; Grif- fiths, 2017). Theories of causation have been stud- ied extensively across philosophy (Beebee et al., 2009), physics (Dowe, 2009), cognitive science (Waldmann, 2017), and probability and statistics (Pearl, 2009), amongst many other fields. Explo- rations of causality in the language domain tend to be semantic, linguistic, or logical in nature as access to direct observational data or event proba- bilities is not assumed nor is required. Descriptions of causality can be linguistically valid but factually incorrect (e.g. butter is the leading cause of factory deaths). Therefore, causal reasoning in language should ideally be logically consistent and grounded Figure 1: An CQA example from the COPA (Gordon et al., 2012). CQA requires identifying causal concepts, linking those concepts to causal relations, and reasoning over those relations. in commonsense knowledge. The counterfactual theory of causation (Lewis, 1973) provides a useful definition of causation for language applications. It posits that causation is relational (there is a cause and effect), temporal (the cause must precede the effect), and counterfactual (if the causing event had not occurred, the effect would not have occurred). Various natural language processing (NLP) appli- cations require identifying causal relations and rea- soning over those relations. These NLP applications can be split into two gen- eral categories: causal relation identification (CRI) and causal question-answering (CQA). CRI tasks aim to identify and extract cause/effect spans from descriptions of causal events. CRI requires lin- guistic knowledge - relying on lexical triggers (i.e. causative verbs and causal connectives) and gram- matical structures (Neeleman et al., 2012; Girju, 2003). …

Content truncated to 500 words