MVP-Tuning: Multi-View Knowledge Retrieval with Prompt Tuning for Commonsense Reasoning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13417–13432 July 9-14, 2023 ©2023 Association for Computational Linguistics MVP-Tuning: Multi-View Knowledge Retrieval with Prompt Tuning for Commonsense Reasoning Yongfeng Huang1, Yanyang Li1,2, Yicong Xu4, Lin Zhang3, Ruyi Gan3, Jiaxing Zhang3, Liwei Wang1∗ 1Department of Computer Science and Engineering, The Chinese University of Hong Kong 2SenseTime Group Inc. 3International Digital Economy Academy (IDEA), China 4Microsoft Cognitive Services Research {yfhuang22, yyli21, lwwang}@cse.cuhk.edu.hk yicxu@microsoft.com,{zhanglin, ganruyi, zhangjiaxing}@idea.edu.cn Abstract Recent advances in pre-trained language mod- els (PLMs) have facilitated the development of commonsense reasoning tasks. However, ex- isting methods rely on multi-hop knowledge retrieval and thus suffer low accuracy due to embedded noise in the acquired knowledge. In addition, these methods often attain high com- putational costs and nontrivial knowledge loss because they encode the knowledge indepen- dently of the PLM, making it less relevant to the task and resulting in a poor local optimum. In this work, we propose Multi-View Knowledge Retrieval with Prompt Tuning (MVP-Tuning). Our MVP-Tuning leverages similar question- answer pairs in training set to improve knowl- edge retrieval and employs a single prompt- tuned PLM to model knowledge and input text jointly. We conduct our experiments on five commonsense reasoning QA benchmarks to show that MVP-Tuning outperforms all other baselines in 4 out of 5 datasets with only as most 2% trainable parameters. The ensemble of our MVP-Tuning models even gets a new state-of-the-art performance on OpenBookQA and is ranked first place on the leaderboard1. Our code and data are available2. 1 Introduction Endowing machines with human-like common- sense reasoning capabilities has gained increas- ing interest in natural language processing in re- cent years (Talmor et al., 2019; Rajpurkar et al., 2018). Large pre-trained language models (De- vlin et al., 2018; Radford et al., 2019; Yang et al., 2019; Brown et al., 2020a; Roberts et al., 2020; He et al., 2020) offer unprecedented potential to mine ∗Corresponding author. 1The anonymous submission is in https:// leaderboard.allenai.org/open_book_qa/submission/ cdtvnvg4kc1nql1dnu3g 2https://github.com/kochsnow/MVP-Tuning/ knowledge because of their unique capability in in- context learning. However, given their black-box nature, these models lack essential interpretability, resulting in the embedded knowledge that is al- ways implicit, difficult to interpret, and fragmented. Therefore, people have developed methods to ex- plicitly inject external knowledge, such as knowl- edge graphs (KG), as contextual knowledge into downstream tasks like commonsense reasoning. The main challenge of the above solution lies in utilizing knowledge to serve individual queries while suffering the scalability issue since there can be millions of nodes in the knowledge graph. Intu- itively, how to extract a partial knowledge graph, i.e., a subgraph, effectively and efficiently is cru- cial. Recent efforts focus on the multi-hop knowl- edge retrieval strategy (Feng et al., 2020a), which anchors input context entities to KG nodes and obtains relevant subgraphs from these nodes and the corresponding multi-hop neighbors. Knowl- edge triplets retrieved by multi-hop retrieval need to be directly connected in the knowledge graph and form a path. This process …

Content truncated to 500 words