Taming Pre-trained Language Models with N-gram Representations for Low-Resource Domain Adaptation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3336–3349 August 1–6, 2021. ©2021 Association for Computational Linguistics 3336 Taming Pre-trained Language Models with N-gram Representations for Low-Resource Domain Adaptation Shizhe Diao♦, Ruijia Xu♦, Hongjin Su♣, Yilei Jiang♣ Yan Song♠♥, Tong Zhang♦ ♦The Hong Kong University of Science and Technology {sdiaoaa, rxuaq, tongzhang}@ust.hk ♣The Chinese University of Hong Kong ♠The Chinese University of Hong Kong (Shenzhen) ♥Shenzhen Research Institute of Big Data songyan@cuhk.edu.cn Abstract Large pre-trained models such as BERT are known to improve different downstream NLP tasks, even when such a model is trained on a generic domain. Moreover, recent studies have shown that when large domain-specific corpora are available, continued pre-training on domain-specific data can further improve the performance of in-domain tasks. How- ever, this practice requires significant domain- specific data and computational resources which may not always be available. In this paper, we aim to adapt a generic pretrained model with a relatively small amount of domain-specific data. We demonstrate that by explicitly incorporating the multi-granularity information of unseen and domain-specific words via the adaptation of (word based) n- grams, the performance of a generic pretrained model can be greatly improved. Specifically, we introduce a Transformer-based Domain- aware N-gram Adaptor, T-DNA, to effectively learn and incorporate the semantic represen- tation of different combinations of words in the new domain. Experimental results illus- trate the effectiveness of T-DNA on eight low- resource downstream tasks from four domains. We show that T-DNA is able to achieve sig- nificant improvements compared to existing methods on most tasks using limited data with lower computational costs. Moreover, further analyses demonstrate the importance and ef- fectiveness of both unseen words and the in- formation of different granularities.1 1 Introduction Pre-trained language models have achieved great success and shown promise in various applica- tion scenarios across natural language understand- ing (Devlin et al., 2019; Liu et al., 2019; Tian et al., 2020a) and generation (Lewis et al., 2020; Zhang 1Our code is available at https://github.com/ shizhediao/T-DNA. et al., 2020; Yang et al., 2020). Normally applying pre-trained language models to different applica- tions follows a two-stage paradigm: pre-training on a large unlabeled corpus and then fine-tuning on a downstream task dataset. However, when there are domain gaps between pre-training and fine-tuning data, previous studies (Beltagy et al., 2019; Lee et al., 2020) have observed a performance drop caused by the incapability of generalization to new domains. Towards filling the gaps, the main re- search stream (Beltagy et al., 2019; Alsentzer et al., 2019; Huang et al., 2019; Lee et al., 2020) on adapting pre-trained language models starts from a generic model (e.g., BERT, RoBERTa) and then continues pre-training with similar objectives on a large-scale domain-specific corpus. However, without providing sufficient understanding of the reason for the performance drop during the domain shift, it is prone to failure of adaptation. There- fore, many aspects of continuous pre-training are expected to be enhanced. …

Content truncated to 500 words