Not to Overfit or Underfit the Source Domains? An Empirical Study of Domain Generalization in Question Answering

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3752 - 3761 December 7-11, 2022 ©2022 Association for Computational Linguistics Not to Overfit or Underfit the Source Domains? An Empirical Study of Domain Generalization in Question Answering Md Arafat Sultan Avirup Sil Radu Florian IBM Research AI arafat.sultan@ibm.com, {avi, raduf}@us.ibm.com Abstract Machine learning models are prone to overfit- ting their training (source) domains, which is commonly believed to be the reason why they falter in novel target domains. Here we exam- ine the contrasting view that multi-source do- main generalization (DG) is first and foremost a problem of mitigating source domain underfit- ting: models not adequately learning the signal already present in their multi-domain training data. Experiments on a reading comprehension DG benchmark show that as a model learns its source domains better—using familiar meth- ods such as knowledge distillation (KD) from a bigger model—its zero-shot out-of-domain utility improves at an even faster pace. Im- proved source domain learning also demon- strates superior out-of-domain generalization over three popular existing DG approaches that aim to limit overfitting. Our implemen- tation of KD-based domain generalization is available via PrimeQA at: https://ibm.biz/ domain-generalization-with-kd. 1 Introduction Domain generalization (DG) seeks to train mod- els on a small number of source domains in a way that maximizes their zero-shot out-of-domain (OOD) utility (Blanchard et al., 2011; Muandet et al., 2013). Many existing DG methods are rooted in the premise that weak generalization under do- main shift occurs due to “source domain over- fitting”: the learning of spurious source domain correlations, i.e., noise, that are unrelated to the core learning task, and are therefore unlikely to be present in novel target domains. The popular domain-invariant learning (DIL) paradigm, for ex- ample, proposes to limit overfitting by imposing some form of cross-domain regularization on the empirical risk minimization (ERM) process over labeled multi-domain training data (Wang et al., 2021; Zhou et al., 2021). We argue in this paper that this emphasis on not fitting the noise minimizes the importance of actually learning the signal present in the source domains, i.e., not underfitting them, for DG. Un- derfitting occurs when a trained model, due to in- adequacies in its capacity or the training procedure, fails to learn training set patterns that are truly rep- resentative of the task. In a recent study of DG in computer vision, Gulrajani and Lopez-Paz (2021) find that large models with sufficiently high ca- pacity can demonstrate strong OOD generalization even with ordinary ERM training—without the need for DIL—when properly configured. Here we ask a very similar question, but (a) in the NLP context of question answering (QA), and (b) for relatively small models that are more prone to underfitting, e.g., a BERT-base QA model (Devlin et al., 2019). Concretely, we investigate if enabling smaller QA models to learn their source domains better also improves their generalization to new domains and how the results compare to DIL. This formulation essentially views OOD gener- alization in low-capacity models as a …

Content truncated to 500 words