To Adapt or to Annotate: Challenges and Interventions for Domain Adaptation in Open-Domain Question Answering

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14429–14446 July 9-14, 2023 ©2023 Association for Computational Linguistics To Adapt or to Annotate: Challenges and Interventions for Domain Adaptation in Open-Domain Question Answering Dheeru Dua1∗ Emma Strubell2∗ Sameer Singh1 Pat Verga3 1University of California, Irvine 2 Carnegie Mellon University 3 Google DeepMind Abstract Recent advances in open-domain question an- swering (ODQA) have demonstrated impres- sive accuracy on general-purpose domains like Wikipedia. While some work has been investi- gating how well ODQA models perform when tested for out-of-domain (OOD) generalization, these studies have been conducted only under conservative shifts in data distribution and typ- ically focus on a single component (i.e., re- triever or reader) rather than an end-to-end sys- tem. This work proposes a more realistic end- to-end domain shift evaluation setting covering five diverse domains. We not only find that end- to-end models fail to generalize but that high retrieval scores often still yield poor answer pre- diction accuracy. To address these failures, we investigate several interventions, in the form of data augmentations, for improving model adaption and use our evaluation set to elucidate the relationship between the efficacy of an in- tervention scheme and the particular type of dataset shifts we consider. We propose a gen- eralizability test that estimates the type of shift in a target dataset without training a model in the target domain and that the type of shift is predictive of which data augmentation schemes will be effective for domain adaption. Over- all, we find that these interventions increase end-to-end performance by up to ∼24 points. 1 Introduction General-purpose open-domain question answer- ing (ODQA; Chen et al. (2017); Lee et al. (2019); Izacard et al. (2022)) is an important task that auto- mates reading and understanding a large corpus of documents to answer a given question succinctly. It is especially crucial in fields such as biomedicine, legal, news, etc., where more documents are added daily, outpacing the speed at which a user can process the information. Current state-of-the-art ODQA systems perform a two-stage pipeline pro- cess (Izacard et al., 2022): 1) Given a question ∗*This work was done while authors were at Google. Average F1 over all target datasets Average F1 over target datasets with specific shifts Figure 1: Effect of interventions on dataset shifts. Top: Average end-to-end performance of source do- main model is quite poor when applied to OOD datasets. Source model (trained on general-purpose domain) per- formance improves when adapted to unseen target do- main with interventions. Bottom: Drill-down of perfor- mance into zero and few-shot data augmentations aver- aged over target datasets exhibiting these shifts shows covariate and concept shifts respond to zero and few- shot data augmentations. Target datasets with No shift do not improve much with any intervention while full shift benefits most from Few-shot. and document corpus, a retriever (Karpukhin et al., 2020; Izacard et al., 2021; Raffel et al., 2020) se- lects relevant passages and 2) a question …

Content truncated to 500 words