Multi-Stage Pre-training for Low-Resource Domain Adaptation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5461–5468, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5461 Multi-Stage Pre-training for Low-Resource Domain Adaptation Rong Zhang∗, Revanth Gangi Reddy∗†, Md Arafat Sultan, Vittorio Castelli, Anthony Ferritto, Radu Florian, Efsun Sarioglu Kayi, Salim Roukos, Avirup Sil‡, Todd Ward IBM Research AI {zhangr,avi,raduf,roukos,toddward,vittorio}@us.ibm.com, g.revanthreddy111@gmail.com, {arafat.sultan,aferritto}@ibm.com, efsun@gwu.edu Abstract Transfer learning techniques are particularly useful in NLP tasks where a sizable amount of high-quality annotated data is difficult to ob- tain. Current approaches directly adapt a pre- trained language model (LM) on in-domain text before fine-tuning to downstream tasks. We show that extending the vocabulary of the LM with domain-specific terms leads to fur- ther gains. To a bigger effect, we utilize struc- ture in the unlabeled data to create auxiliary synthetic tasks, which helps the LM transfer to downstream tasks. We apply these approaches incrementally on a pre-trained Roberta-large LM and show considerable performance gain on three tasks in the IT domain: Extractive Reading Comprehension, Document Ranking and Duplicate Question Detection. 1 Introduction Pre-trained language models (Radford et al., 2019; Devlin et al., 2019; Liu et al., 2019) have pushed performance in many natural language processing tasks to new heights. The process of model con- struction has effectively been reduced to extending the pre-trained LM architecture with simpler task- specific layers, while fine-tuning on labeled target data. In cases where the target task has limited labeled data, prior work has also employed trans- fer learning by pre-training on a source dataset with abundant labeled data before fine-tuning on the target task dataset (Min et al., 2017; Chung et al., 2018; Wiese et al., 2017). However, directly fine-tuning to a task in a new domain may not be optimal when the domain is distant in content and terminology from the pre-training corpora. To address this language mismatch problem, re- cent work (Alsentzer et al., 2019; Lee et al., 2019; ∗ Both authors contributed equally. † Work done during AI Residency at IBM Research. ‡ Corresponding author. Beltagy et al., 2019; Gururangan et al., 2020) has adapted pre-trained LMs to specific domains by continuing to train the same LM on target do- main text. Similar approaches are also used in multilingual adaptation, where the representations learned from multilingual pre-training are further optimized for a particular target language (Liu et al., 2020; Bapna and Firat, 2019). However, many spe- cialized domains contain their own specific terms that are not part of the pre-trained LM vocabulary. Furthermore, in many such domains, large enough corpora may not be available to support LM train- ing from scratch. To resolve this out-of-vocabulary issue, in this work, we extend the open-domain vo- cabulary with in-domain terms while adapting the LM, and show that it helps improve performance on downstream tasks. While language modeling can help the model better encode the domain language, it might not be sufficient to gain the domain knowledge neces- sary for the downstream task. We remark, how- ever, that such unlabeled data …

Content truncated to 500 words