Preserving Commonsense Knowledge from Pre-trained Language Models via Causal Inference

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 9155–9173 July 9-14, 2023 ©2023 Association for Computational Linguistics Preserving Commonsense Knowledge from Pre-trained Language Models via Causal Inference Junhao Zheng, Qianli Ma*, Shengjie Qiu, Yue Wu, Peitian Ma, Junlong Liu, Huawen Feng, Xichen Shang and Haibin Chen School of Computer Science and Engineering, South China University of Technology, Guangzhou, China junhaozheng47@outlook.com, qianlima@scut.edu.cn∗ Abstract Fine-tuning has been proven to be a simple and effective technique to transfer the learned knowledge of Pre-trained Language Models (PLMs) to downstream tasks. However, vanilla fine-tuning easily overfits the target data and degrades the generalization ability. Most exist- ing studies attribute it to catastrophic forgetting, and they retain the pre-trained knowledge in- discriminately without identifying what knowl- edge is transferable. Motivated by this, we frame fine-tuning into a causal graph and dis- cover that the crux of catastrophic forgetting lies in the missing causal effects from the pre- trained data. Based on the causal view, we propose a unified objective for fine-tuning to retrieve the causality back. Intriguingly, the unified objective can be seen as the sum of the vanilla fine-tuning objective, which learns new knowledge from target data, and the causal ob- jective, which preserves old knowledge from PLMs. Therefore, our method is flexible and can mitigate negative transfer while preserving knowledge. Since endowing models with com- monsense is a long-standing challenge, we im- plement our method on commonsense QA with a proposed heuristic estimation to verify its ef- fectiveness. In the experiments, our method outperforms state-of-the-art fine-tuning meth- ods on all six commonsense QA datasets and can be implemented as a plug-in module to in- flate the performance of existing QA models. 1 1 Introduction Deep Pre-trained Language Models (PLMs) such as RoBERTa (Liu et al., 2019b) and T5 (Raffel et al., 2020)) are inherently knowledge bases since they are exposed to a tremendous amount of data (e.g., the C4 dataset (Raffel et al., 2020)) in the ∗*Corresponding author 1Our codes are publicly available at https://github.com/zzz47zzz/CET and https://github.com/qianlima-lab/CET pre-training stage (Petroni et al., 2019; AlKhamissi et al., 2022). Unfortunately, transferring the intrin- sic knowledge in PLMs to downstream tasks is non- trivial. In practice, fine-tuning is adopted widely due to its flexibility (Chen et al., 2020) and numer- ous improved methods (Lee et al., 2019; Chen et al., 2020, 2019; Mosbach et al., 2020; Zhang et al., 2020b; Xu et al., 2021a; Aghajanyan et al., 2020; Wu et al., 2022) are proposed in recent years. How- ever, fine-tuning faces two challenges when adapt- ing models to new domains (Chen et al., 2019), including catastrophic forgetting (Kirkpatrick et al., 2017) and negative transfer (Torrey and Shavlik, 2010). More specifically, catastrophic forgetting refers to models losing previously learned knowl- edge and overfitting the target domain data. Neg- ative transfer occurs because not all pre-trained knowledge is transferable across domains. Obvi- ously, catastrophic forgetting and negative transfer constitute a dilemma where the crux lies in identi- fying and utilizing …

Content truncated to 500 words