Knowledgeable Parameter Efficient Tuning Network for Commonsense Question Answering

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 9051–9063 July 9-14, 2023 ©2023 Association for Computational Linguistics Knowledgeable Parameter Efficient Tuning Network for Commonsense Question Answering Ziwang Zhao1 Linmei Hu2∗ Hanyu Zhao3 Yingxia Shao1 Yequan Wang3 1Beijing University of Posts and Telecommunications 2 Beijing Institute of Technology 3Beijing Academy of Artificial Intelligence {zhaoziwang,shaoyx}@bupt.edu.cn hulinmei@bit.edu.cn hyzhao@baai.ac.cn tshwangyequan@gmail.com Abstract Commonsense question answering is important for making decisions about everyday matters. Although existing commonsense question an- swering works based on fully fine-tuned PLMs have achieved promising results, they suffer from prohibitive computation costs as well as poor interpretability. Some works improve the PLMs by incorporating knowledge to provide certain evidence, via elaborately designed GNN modules which require expertise. In this paper, we propose a simple knowledgeable parameter efficient tuning network to couple PLMs with external knowledge for commonsense question answering. Specifically, we design a train- able parameter-sharing adapter attached to a parameter-freezing PLM to incorporate knowl- edge at a small cost. The adapter is equipped with both entity- and query-related knowledge via two auxiliary knowledge-related tasks (i.e., span masking and relation discrimination). To make the adapter focus on the relevant knowl- edge, we design gating and attention mecha- nisms to respectively filter and fuse the query information from the PLM. Extensive exper- iments on two benchmark datasets show that KPE is parameter-efficient and can effectively incorporate knowledge for improving common- sense question answering. 1 Introduction Commonsense question answering is the process of combining observations and the basic knowl- edge that reflects our natural understanding of the world and human behaviors, to make presumptions about ordinary situations in our daily life (Johnson- Laird, 1980). It has emerged as an important task in natural language understanding. Pre-trained language models (PLMs), which revolutionize many areas with superior perfor- mance, have been applied for the commonsense question answering task based on full fine-tuning ∗Corresponding author Knowledgeable Adapter (Trainable) Existing Methods Our Proposed KPE Gating Grounding Scorer Method (a) Cotton candy is sometimes made out of cotton. Query: A: yes B: no* Answer: Method (b) Derived From MadeOf IsA HasA Knowledge Subgraph Input Input Knowledge GNN Based Module Pre-trained Language Models (Fully Fine-tuning) Scorer Pre-trained Language Models (Freezing) Scorer Serializing & Grounding Input Span masking Relation Discrimination Cotton candy Sugar Candy [Query, Answer] Cotton [Query, Answer] Knowledge Unstructured GenericsKB Figure 1: Comparison of existing methods and our pro- posed method. as shown in Figure 1(a). For example, Lourie et al. (2021) fully fine-tuned the PLM Unicorn and achieved competitive performance on 8 common- sense benchmarks. However, they inevitably incur prohibitive computation costs as the scale of param- eters increases, and are lacking in transparency and interpretability (Houlsby et al., 2019a; Lin et al., 2019). Furthermore, some works couple the PLMs with knowledge to improve the interpretability of the reasoning process. As shown in Figure 1(b), they typically extract the relevant knowledge subgraphs about entities in the query and then elaborately design a graph neural network (GNN) module to perform …

Content truncated to 500 words