QA Domain Adaptation using Hidden Space Augmentation and Self-Supervised Contrastive Adaptation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2308 - 2321 December 7-11, 2022 ©2022 Association for Computational Linguistics QA Domain Adaptation using Hidden Space Augmentation and Self-Supervised Contrastive Adaptation Zhenrui Yue∗ UIUC zhenrui3@illinois.edu Huimin Zeng∗ UIUC huiminz3@illinois.edu Bernhard Kratzwald EthonAI bernhard.kratzwald@ethon.ai Stefan Feuerriegel LMU Munich feuerriegel@lmu.de Dong Wang UIUC dwang24@illinois.edu Abstract Question answering (QA) has recently shown impressive results for answering questions from customized domains. Yet, a common challenge is to adapt QA models to an unseen target do- main. In this paper, we propose a novel self- supervised framework called QADA for QA domain adaptation. QADA introduces a novel data augmentation pipeline used to augment training QA samples. Different from exist- ing methods, we enrich the samples via hid- den space augmentation. For questions, we introduce multi-hop synonyms and sample aug- mented token embeddings with Dirichlet distri- butions. For contexts, we develop an augmenta- tion method which learns to drop context spans via a custom attentive sampling strategy. Addi- tionally, contrastive learning is integrated in the proposed self-supervised adaptation framework QADA. Unlike existing approaches, we gener- ate pseudo labels and propose to train the model via a novel attention-based contrastive adapta- tion method. The attention weights are used to build informative features for discrepancy estimation that helps the QA model separate answers and generalize across source and tar- get domains. To the best of our knowledge, our work is the first to leverage hidden space aug- mentation and attention-based contrastive adap- tation for self-supervised domain adaptation in QA. Our evaluation shows that QADA achieves considerable improvements on multiple target datasets over state-of-the-art baselines in QA domain adaptation. 1 Introduction Question answering (QA) is the task of finding answers for a given context and a given question. QA models are typically trained using data triplets consisting of context, question and answer. In the case of extractive QA, answers are represented as subspans in the context defined by a start position and an end position, while question and context are ∗Both authors contributed equally to this research. given as running text (e.g., Seo et al., 2016; Chen et al., 2017; Devlin et al., 2019; Kratzwald et al., 2019). A common challenge in extractive QA is that QA models often suffer from performance deterioration upon deployment and thus make mistakes for user- generated inputs. The underlying reason for such deterioration can be traced back to the domain shift between training data (from the source domain) and test data (from the target domain) (Fisch et al., 2019; Miller et al., 2020; Zeng et al., 2022b). Existing approaches to address domain shifts in extractive QA can be grouped as follows. One approach is to include labeled target examples or user feedback during training (Daumé III, 2007; Kratzwald and Feuerriegel, 2019a; Kratzwald et al., 2020; Kamath et al., 2020). Another approach is to generate labeled QA samples in the target domain for training (Lee et al., 2020; Yue et al., 2021a, 2022a). However, these approaches typically re- quire large …

Content truncated to 500 words