Efficient Hierarchical Domain Adaptation for Pretrained Language Models

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1336 - 1351 July 10-15, 2022 ©2022 Association for Computational Linguistics Efficient Hierarchical Domain Adaptation for Pretrained Language Models Alexandra Chronopoulou1∗, Matthew E. Peters2, Jesse Dodge2 1Center for Information and Language Processing, LMU Munich, Germany 2Allen Institute for Artificial Intelligence, Seattle, WA achron@cis.lmu.de {matthewp,jessed}@allenai.org Abstract The remarkable success of large language models has been driven by dense models trained on massive unlabeled, unstructured cor- pora. These corpora typically contain text from diverse, heterogeneous sources, but infor- mation about the source of the text is rarely used during training. Transferring their knowl- edge to a target domain is typically done by continuing training in-domain. In this paper, we introduce a method to permit domain adap- tation to many diverse domains using a com- putationally efficient adapter approach. Our method is based on the observation that tex- tual domains are partially overlapping, and we represent domains as a hierarchical tree structure where each node in the tree is asso- ciated with a set of adapter weights. When combined with a frozen pretrained language model, this approach enables parameter shar- ing among related domains, while avoiding negative interference between unrelated ones. Experimental results with GPT-2 and a large fraction of the 100 most represented websites in C4 show across-the-board improvements in- domain. We additionally provide an inference time algorithm for a held-out domain and show that averaging over multiple paths through the tree enables further gains in generalization, while adding only a marginal cost to inference. 1 Introduction Pretrained language models (PLMs) (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019; Radford et al., 2019), trained on massive general-domain corpora, have enabled great progress in many natu- ral language processing (NLP) benchmarks (Wang et al., 2018). Nonetheless, continuing pretraining (as a dense model) a PLM on a narrower domain (Han and Eisenstein, 2019; Lee et al., 2019) is ben- eficial, although computationally expensive (Ma- ronikolakis and Schütze, 2021), which indicates ∗Work done while an intern at AllenAI. 6 3 4 scientific articles reviews frontiersin.org journals.plos.org 7 booking.com 5 1 2 yelp.com Figure 1: We model domains as a hierarchical tree structure that associates adapters with nodes, allowing parameter sharing among related domains. Internet do- mains appear as leaf nodes. During training, we acti- vate the adapters along the path to a leaf to specialize a language model to the domain corresponding to it. that domain-relevant data is important for down- stream tasks. Sparse models that use mixtures of experts (Lepikhin et al., 2021) have recently been proposed to allow efficient training. Prior work typically assumes that individual do- mains are distinct, and models them accordingly. For example, Gururangan et al. (2020, 2021) train one model for each textual domain, either in a dense or sparse manner. This is related to data selection (Moore and Lewis, 2010; Axelrod et al., 2011; Plank and van Noord, 2011), which aims to select the …

Content truncated to 500 words