Momentum Contrastive Pre-training for Question Answering

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4324 - 4330 December 7-11, 2022 ©2022 Association for Computational Linguistics Momentum Contrastive Pre-training for Question Answering Minda Hu1 Muzhi Li1 Yasheng Wang2 Irwin King1 1Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 2Huawei Noah’s Ark Lab {mindahu21, mzli, king}@cse.cuhk.edu.hk, wangyasheng@huawei.com Abstract Existing pre-training methods for extractive Question Answering (QA) generate cloze-like queries different from natural questions in syn- tax structure, which could overfit pre-trained models to simple keyword matching. In or- der to address this problem, we propose a novel Momentum Contrastive pRe-training fOr queStion anSwering (MCROSS) method for ex- tractive QA. Specifically, MCROSS introduces a momentum contrastive learning framework to align the answer probability between cloze-like and natural query-passage sample pairs. Hence, the pre-trained models can better transfer the knowledge learned in cloze-like samples to an- swering natural questions. Experimental results on three benchmarking QA datasets show that our method achieves noticeable improvement compared with all baselines in both supervised and zero-shot scenarios. 1 Introduction The task of extractive Question Answering (QA), which aims to select an answer span from a pas- sage given a query, is a major focus of NLP re- search. Currently, deep learning systems (Huang et al., 2018; Devlin et al., 2019; Wu et al., 2021) have achieved competitive results with humans on large-scale QA datasets (Rajpurkar et al., 2016, 2018; Yang et al., 2018; Kwiatkowski et al., 2019). However, the collection of high-quality natural QA pairs is still a labor-intensive task, especially for the construction of domain-specific QA systems. To alleviate such data availability restrictions, pre- training methods have been drawing increasing at- tention (Dhingra et al., 2018; Ram et al., 2021). Typically, pre-training methods for extractive QA generate cloze-like query-passage pairs with text-matching techniques. For instance, the Span Selection Pre-Training (SSPT) method (Glass et al., 2020a) generates these pairs with the Wikipedia corpus using BM25. Nevertheless, these meth- ods have two major issues. Firstly, the format of In Jordan's second season of the playoffs against the [BLANK], he even scored 63 points Cloze-like Query What team did Jordan score 63 points against in the second season of the playoffs? Natural Query Boston Celtics Answer Term Figure 1: An example of the pre-training queries in SSPT (Glass et al., 2020b). The query is a sentence where a noun phrase or entity (“Boston Celtics”) is replaced with the special token [BLANK], which differs syntactically from the natural one. cloze-like queries differs much from natural queries asked by humans (see Fig. 1). In addition, models trained by cloze-like queries overfit in capturing the lexical overlaps between queries and passages, which restricts the capability of high-level semantic reasoning (Li et al., 2020; Hu et al., 2021). Since answering natural questions requires a comprehen- sive understanding of queries and passages, mod- els pre-trained with cloze-like queries are not well aligned with downstream QA tasks. To solve these issues, we utilize Contrastive Learning (CL) techniques to align knowledge learned in cloze-like …

Content truncated to 500 words