The Trade-offs of Domain Adaptation for Neural Language Models

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3802 - 3813 May 22-27, 2022 c⃝2022 Association for Computational Linguistics The Trade-offs of Domain Adaptation for Neural Language Models David Grangier Google, Mountain View, CA grangier@google.com Dan Iter∗ Stanford, Palo Alto, CA daniter@stanford.edu Abstract This work connects language model adapta- tion with concepts of machine learning theory. We consider a training setup with a large out- of-domain set and a small in-domain set. We derive how the benefit of training a model on either set depends on the size of the sets and the distance between their underlying distri- butions. We analyze how out-of-domain pre- training before in-domain fine-tuning achieves better generalization than either solution inde- pendently. Finally, we present how adapta- tion techniques based on data selection, such as importance sampling, intelligent data selec- tion and influence functions, can be presented in a common framework which highlights their similarity and also their subtle differences. 1 Introduction Neural Language Models (LMs) trained on large generic training sets – over a billion sen- tences (Kaplan et al., 2020; Roziewski and Kozłowski, 2021) – have been shown to be ef- fective at adapting to smaller, specific target do- mains for language modeling and other down- stream tasks (Bommasani et al., 2021). Neural LM adaptation is commonly performed via fine tun- ing (Devlin et al., 2018; Liu et al., 2019; Raffel et al., 2019; Radford et al., 2019), data selection (van der Wees et al., 2017) or their combination (Wang et al., 2018; Aharoni and Goldberg, 2020; Gururangan et al., 2020). However, the trade- offs between fine-tuning and reweighting of pre- training data is not well understood and a theoreti- cal framework for reasoning about the generaliza- tion performance of these methods is needed. In this paper, we connect language model adap- tation with concepts of machine learning theory. Our derivations support past empirical observa- tions: it has been observed that the size of the out-of-domain pre-training set is important for in ∗Work performed while interning at Google. domain generalization (Raffel et al., 2019; Devlin et al., 2018) or that domain adaptation is more ef- fective on domains which are well represented in the the pre-training data (Radford et al., 2019). Our study consider a training setup with a large out-of-domain set and a small in-domain set. As a first contribution, we derive how the benefit of training a model on either set depends on the size of the sets and the distance between their underly- ing distribution. We also expose how fine-tuning can be viewed as a regularization method that can achieve a better trade-off than training only on ei- ther set. The research on data selection for LM adap- tion originates mainly from intelligent selec- tion (Moore and Lewis, 2010; Axelrod et al., 2011). This method examines the out-of-domain training data to emphasize a subset deemed more likely by an in-domain model than by an out-of- domain model. Although intuitive, the connection of …

Content truncated to 500 words