Impact of Evaluation Methodologies on Code Summarization

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 432 - 446 May 22-27, 2022 c⃝2022 Association for Computational Linguistics e-CARE: a New Dataset for Exploring Explainable Causal Reasoning Li Du, Xiao Ding∗, Kai Xiong, Ting Liu, and Bing Qin Research Center for Social Computing and Information Retrieval Harbin Institute of Technology, China {ldu, xding, kxiong, tliu,qinb}@ir.hit.edu.cn Abstract Understanding causality has vital importance for various Natural Language Processing (NLP) applications. Beyond the labeled in- stances, conceptual explanations of the causal- ity can provide deep understanding of the causal facts to facilitate the causal reasoning process. However, such explanation informa- tion still remains absent in existing causal rea- soning resources. In this paper, we fill this gap by presenting a human-annotated explainable CAusal REasoning dataset (e-CARE), which contains over 21K causal reasoning questions, together with natural language formed expla- nations of the causal questions. Experimental results show that generating valid explanations for causal facts still remains especially chal- lenging for the state-of-the-art models, and the explanation information can be helpful for promoting the accuracy and stability of causal reasoning models. 1 Introduction Causal reasoning is one of the most central cog- nitive abilities of human beings (Waldmann and Hagmayer, 2013; Jonassen et al., 2008), which en- ables one to understand the observed facts and pre- dict the future. However, although recent causal reasoning models have achieved impressive per- formances on certain hand-crafted datasets, there still remains a considerable gap compared to hu- man performances, as they cannot achieve stable performances across different datasets and are sus- ceptible to adversarial attacks (McCoy et al., 2019; Poliak et al., 2018; Gururangan et al., 2018). One key factor leading to such drastic contrast is that, present causal reasoning models only learn to induce empirical causal patterns that are predic- tive to the label, while human beings seek for deep and conceptual understanding of the causality to explain the observed causal facts. The conceptual ∗Corresponding author Cause Effect Causal Relationship Conceptual Explanation Observed Causal Fact C1: Adding rock into hydrochloric acid E1: Rock dissolved Acid is corrosive Figure 1: Conceptual explanations of observed causality can be helpful for understanding the unseen causal facts. explanations can not only serve as a touchstone to examine whether the underlying causal mech- anism has been thoroughly understood, but it can also in turn support the causal reasoning process. As illustrated in Figure 1, observing the causal fact C1: adding rock into hydrochloric acid causes E1: rock dissolved, one may further ask why such a causal relationship exists and reach the plausi- ble conceptual explanation that Acid is corrosive, which goes beyond the isolated facts and reaches the conceptual nature to reveal the principle of the causal mechanism. However, despite the critical importance of con- ceptual explanations in causal reasoning, there is still a lack of such an explainable causal rea- soning dataset. To fill this gap, we contribute an explainable CAusal REasoning dataset (e- CARE),together with a new causal explanation generation …

Content truncated to 500 words