Integrating Task Specific Information into Pretrained Language Models for Low Resource Fine Tuning

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3181–3186 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3181 Integrating Task Specific Information into Pretrained Language Models for Low Resource Fine Tuning Rui Wang1* Shijing Si1* Guoyin Wang1,2 Lei Zhang3 Lawrence Carin1 Ricardo Henao1 1Duke University 2Amazon Alexa AI 3Fidelity Investments rui.wang16@duke.edu Abstract Pretrained Language Models (PLMs) have im- proved the performance of natural language understanding in recent years. Such mod- els are pretrained on large corpora, which encode the general prior knowledge of natu- ral languages but are agnostic to information characteristic of downstream tasks. This of- ten results in overfitting when fine-tuned with low resource datasets where task-specific in- formation is limited. In this paper, we inte- grate label information as a task-specific prior into the self-attention component of pretrained BERT models. Experiments on several bench- marks and real-word datasets suggest that the proposed approach can largely improve the performance of pretrained models when fine- tuning with small datasets. The code repos- itory is released in https://github.com/ RayWangWR/BERT_label_embedding. 1 Introduction Recently, Pretrained Language Models (PLMs) (Devlin et al., 2018; Radford et al., 2019) have yield significant progress on various natural language processing (NLP) tasks, e.g., neural language un- derstanding, text generation, etc. Existing PLMs are usually pretrained in a task-agnostic manner, in which the model is expected to capture the general knowledge of natural language from a large corpus, independent of downstream-specific information. This is not a problem when data is abundant in the downstream dataset, in which case, the model can effectively extract task-specific information during fine-tuning. However, in real scenarios, data may be difficult to collect and labeling is usually expen- sive. We show that PLMs pretrained with general knowledge can overfit without enough guidance from the task-specific information, resulting in de- graded performance during testing. *These authors contributed equally to this work A clear-cut solution to this problem is to focus more on samples that are more relevant to the tar- get task during pretraining. However, this requires a task-specific pretraining, which in most cases is computational or time prohibitive. Another ap- proach is to pretrain on an auxiliary dataset before fine-tuning on the target task (Phang et al., 2018). Such method requires the availability of an appro- priate auxiliary datasets. Unfortunately, in some cases it may negatively impact the downstream transfer (Wang et al., 2018a). Label embeddings (Akata et al., 2015) can be regarded as a feature- based definition of a classification task, in which detailed information of the task is encoded. One natural question is whether we can combine the general knowledge in a PLM and the task-specific characterization contained within label embeddings for better fine-tuning on low-resource tasks. In this paper, we propose to utilize the label em- beddings as a task-specific prior, complementary to the general prior already encoded during pretrain- ing. We learn and integrate these label embeddings into BERT models (Devlin et al., 2018) to regular- ize its self-attention modules, so the task-irrelevant tokens or patterns …

Content truncated to 500 words