Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8342 Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks Suchin Gururangan† Ana Marasovi´c†♦ Swabha Swayamdipta† Kyle Lo† Iz Beltagy† Doug Downey† Noah A. Smith†♦ †Allen Institute for Artificial Intelligence, Seattle, WA, USA ♦Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA, USA {suching,anam,swabhas,kylel,beltagy,dougd,noah}@allenai.org Abstract Language models pretrained on text from a wide variety of sources form the foundation of today’s NLP. In light of the success of these broad-coverage models, we investigate whether it is still helpful to tailor a pretrained model to the domain of a target task. We present a study across four domains (biomedi- cal and computer science publications, news, and reviews) and eight classification tasks, showing that a second phase of pretraining in- domain (domain-adaptive pretraining) leads to performance gains, under both high- and low-resource settings. Moreover, adapting to the task’s unlabeled data (task-adaptive pretraining) improves performance even after domain-adaptive pretraining. Finally, we show that adapting to a task corpus augmented us- ing simple data selection strategies is an effec- tive alternative, especially when resources for domain-adaptive pretraining might be unavail- able. Overall, we consistently find that multi- phase adaptive pretraining offers large gains in task performance. 1 Introduction Today’s pretrained language models are trained on massive, heterogeneous corpora (Raffel et al., 2019; Yang et al., 2019). For instance, ROBERTA (Liu et al., 2019) was trained on over 160GB of uncom- pressed text, with sources ranging from English- language encyclopedic and news articles, to literary works and web content. Representations learned by such models achieve strong performance across many tasks with datasets of varying sizes drawn from a variety of sources (e.g., Wang et al., 2018, 2019). This leads us to ask whether a task’s textual domain—a term typically used to denote a distribu- tion over language characterizing a given topic or genre (such as “science” or “mystery novels”)—is still relevant. Do the latest large pretrained mod- els work universally or is it still helpful to build Figure 1: An illustration of data distributions. Task data is comprised of an observable task distribution, usually non-randomly sampled from a wider distribu- tion (light grey ellipsis) within an even larger target do- main, which is not necessarily one of the domains in- cluded in the original LM pretraining domain – though overlap is possible. We explore the benefits of contin- ued pretraining on data from the task distribution and the domain distribution. separate pretrained models for specific domains? While some studies have shown the benefit of continued pretraining on domain-specific unlabeled data (e.g., Lee et al., 2019), these studies only con- sider a single domain at a time and use a language model that is pretrained on a smaller and less di- verse corpus than the most recent language mod- els. Moreover, it is not known how the benefit of continued pretraining may vary with factors like …

Content truncated to 500 words