ProQA: Structural Prompt-based Pre-training for Unified Question Answering

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4230 - 4243 July 10-15, 2022 ©2022 Association for Computational Linguistics ProQA: Structural Prompt-based Pre-training for Unified Question Answering Wanjun Zhong1∗, Yifan Gao4∗, Ning Ding3, Yujia Qin3, Zhiyuan Liu3, Ming Zhou5, Jiahai Wang1, Jian Yin1 and Nan Duan2 1 Sun Yat-sen University 2 Microsoft Research Asia 3 Tsinghua University 4 Chinese University of Hong Kong 5 Langboat Technology {zhongwj25@mail2, wangjiah@mail,issjyin@mail}.sysu.edu.cn yfgao@cse.cuhk.edu.hk; liuzy@tsinghua.edu.cn {dingn18, qyj20}@mails.tsinghua.edu.cn nanduan@microsoft.com; zhouming@chuangxin.com Abstract Question Answering (QA) is a longstanding challenge in natural language processing. Ex- isting QA works mostly focus on specific ques- tion types, knowledge domains, or reasoning skills. The specialty in QA research hinders sys- tems from modeling commonalities between tasks and generalization for wider applications. To address this issue, we present ProQA, a unified QA paradigm that solves various tasks through a single model. ProQA takes a unified structural prompt as the bridge and improves the QA-centric ability by structural prompt- based pre-training. Through a structurally de- signed prompt-based input schema, ProQA concurrently models the knowledge generaliza- tion for all QA tasks while keeping the knowl- edge customization for every specific QA task. Furthermore, ProQA is pre-trained with struc- tural prompt-formatted large-scale synthesized corpus, which empowers the model with the commonly-required QA ability. Experimen- tal results on 11 QA benchmarks demonstrate that ProQA consistently boosts performance on both full data fine-tuning, few-shot learning, and zero-shot testing scenarios. Furthermore, ProQA exhibits strong ability in both contin- ual learning and transfer learning by taking the advantages of the structural prompt.1 1 Introduction Question Answering has long been an inspirational challenge in NLP research, and is viewed as the next-generation search engine and an essential tool for human beings to obtain knowledge (Etzioni, 2011). Many distinct datasets (Rajpurkar et al., 2016; Lai et al., 2017; Kwiatkowski et al., 2019; Gao et al., 2021) have been proposed along with the research trend on QA, involving very diverse ∗ Indicates equal contribution 1The code is available at https://github.com/ zhongwanjun/ProQA. question types (e.g., extractive QA, abstractive QA, multiple-choice QA), domains (e.g., finance, daily events), and answer types (e.g., free-formed text, se- lected option). The majority of previous works fo- cus on tasks with specific question types (Lai et al., 2017; Yang et al., 2018; Gao et al., 2020) or spe- cific domains (Trischler et al., 2017; Kwiatkowski et al., 2019). Recent research on large pre-trained language models (Brown et al., 2020; Bommasani et al., 2021) indicates that there may be tight con- nections among various tasks, which sheds light on a unified paradigm that can be potentially applied to solve various QA tasks to model their common- ality. This observation motivates us to develop a unified QA model, which can model both the commonly-required QA ability and the difference between various QA tasks within a same paradigm. To achieve this goal, there are several key chal- lenges needed to be addressed: (1) How to model commonalities and enhance transferability …

Content truncated to 500 words