TADPOLE: Task ADapted Pre-Training via AnOmaLy DEtection

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5732–5746 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5732 TADPOLE: Task ADapted Pre-training via anOmaLy dEtection Vivek Madan AWS AI Labs vivmadan@amazon.com Ashish Khetan AWS AI Labs khetan@amazon.com Zohar Karnin AWS AI Labs zkarnin@amazon.com Abstract The paradigm of pre-training followed by fine- tuning has become a standard procedure for NLP tasks, with a known problem of domain shift between the pre-training and downstream corpus. Previous works have tried to mitigate this problem with additional pre-training, ei- ther on the downstream corpus itself when it is large enough, or on a manually curated un- labeled corpus of a similar domain. In this pa- per, we address the problem for the case when the downstream corpus is too small for addi- tional pre-training. We propose TADPOLE, a task adapted pre-training framework based on data selection techniques adapted from Do- main Adaptation. We formulate the data se- lection as an anomaly detection problem that unlike existing methods works well when the downstream corpus is limited in size. It results in a scalable and efficient unsupervised tech- nique that eliminates the need for any man- ual data curation. We evaluate our frame- work on eight tasks across four different do- mains: Biomedical, Computer Science, News, and Movie reviews, and compare its perfor- mance against competitive baseline techniques from the area of Domain Adaptation. Our framework outperforms all the baseline meth- ods. On small datasets with less than 5K train- ing examples, we get a gain of 1.82% in per- formance with additional pre-training for only 5% steps. It also compliments some of the other techniques such as data augmentation known for boosting performance when down- stream corpus is small; highest performance is achieved when data augmentation is combined with task adapted pre-training. 1 Introduction Pre-trained language models such as ELMo (Peters et al., 2018), GPT (Radford et al., 2018), BERT (Devlin et al., 2018), Transformer-xl (Dai et al., 2019) and XLNet (Yang et al., 2019) have become a key component in solving virtually all natural language tasks. These models are pre-trained on large amount of cross-domain data ranging from Wikipedia to Book corpus to news articles to learn powerful representations. A generic approach for using these models consists of two steps: (a) Pre- training: train the model on an extremely large general domain corpus, e.g. with masked language model loss; (b) Finetuning: finetune the model on labeled task dataset for the downstream task. Even though the approach of pre-training fol- lowed by fine-tuning has been very successful, it suffers from domain shift when applied to tasks con- taining text from a domain that is not sufficiently represented in the pre-training corpus. An immedi- ate way of solving the problem is to pre-train the model on task domain data instead of the general domain data. For a handful of very popular task do- mains, the research community invested time and resources to collect a large domain-specific data corpus and pre-train a …

Content truncated to 500 words