KECP: Knowledge Enhanced Contrastive Prompting for Few-shot Extractive Question Answering

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3152 - 3163 December 7-11, 2022 ©2022 Association for Computational Linguistics KECP: Knowledge Enhanced Contrastive Prompting for Few-shot Extractive Question Answering Jianing Wang1∗, Chengyu Wang2∗, Minghui Qiu2, Qiuhui Shi3, Hongbin Wang3, Jun Huang2, Ming Gao1,4† 1 School of Data Science and Engineering, East China Normal University, Shanghai, China 2 Alibaba Group, Hangzhou, China 3 Ant Group, Hangzhou, China 4 KLATASDS-MOE, School of Statistics, East China Normal University, Shanghai, China lygwjn@gmail.com, chengyu.wcy@alibaba-inc.com minghui.qmh@alibaba-inc.com, qiuhui.sqh@antgroup.com hongbin.whb@@antgroup.com, huangjun.hj@alibaba-inc.com mgao@dase.ecnu.edu.cn Abstract Extractive Question Answering (EQA) is one of the most essential tasks in Machine Reading Comprehension (MRC), which can be solved by fine-tuning the span selecting heads of Pre-trained Language Models (PLMs). How- ever, most existing approaches for MRC may perform poorly in the few-shot learning sce- nario. To solve this issue, we propose a novel framework named Knowledge Enhanced Contrastive Prompt-tuning (KECP). Instead of adding pointer heads to PLMs, we introduce a seminal paradigm for EQA that transforms the task into a non-autoregressive Masked Lan- guage Modeling (MLM) generation problem. Simultaneously, rich semantics from the ex- ternal knowledge base (KB) and the passage context support enhancing the query’s represen- tations. In addition, to boost the performance of PLMs, we jointly train the model by the MLM and contrastive learning objectives. Ex- periments on multiple benchmarks demonstrate that our method consistently outperforms state- of-the-art approaches in few-shot settings by a large margin. 1 1 Introduction Span-based Extractive Question Answering (EQA) is one of the most challenging tasks of Machine Reading Comprehension (MRC). A majority of re- cent approaches (Wang and Jiang, 2019; Yang et al., 2019; Dai et al., 2021) add pointer heads (Vinyals et al., 2015) to Pre-trained Language Models (PLMs) to predict the start and the end positions of the answer span (shown in Figure 1(a)). Yet, these conventional fine-tuning frameworks heavily ∗ J. Wang and C. Wang contributed equally to this work. † Corresponding author. 1All datasets are publicly available. Source codes will be released in EasyNLP (Wang et al., 2022). URL: https: //github.com/alibaba/EasyNLP PLM Embeddings Pointer Heads [MASK] [MASK] [MASK] (a) Standard Fine-tuning Framework for EQA (b) The Prompt-tuning Framework for EQA PLMs PLM Embeddings [End] ... fighting ... horsemen Passage: ... Normandy had been exporting fighting horsemen for more than a generation. Many Normans of Italy, France and England eventually ... Query: What was one of the Norman's major exports? Passage: ... Normandy had been exporting fighting horsemen for more than a generation. Many Normans of Italy, France and England eventually ... Query: [MASK] [MASK] [MASK] was one of the Norman's major exports. start position end position PLMs MLM Heads ... ... ... ... Figure 1: The comparison of the standard fine-tuning and prompt-tuning framework. The blocks in orange and green denote the modules of PLMs and newly ini- tialized modules, respectively. (Best viewed in color.) depend on the time-consuming and labor-intensive process of data annotation. Additionally, there is a large gap between the pre-training objective …

Content truncated to 500 words