Meta Distant Transfer Learning for Pre-trained Language Models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9742–9752 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9742 Meta Distant Transfer Learning for Pre-trained Language Models Chengyu Wang1,2, Haojie Pan2, Minghui Qiu2∗, Fei Yang1, Jun Huang2, Yin Zhang3 1 Zhejiang Lab, 2 Alibaba Group 3 College of Computer Science and Technology, Zhejiang University {chengyu.wcy,haojie.phj,minghui.qmh}@alibaba-inc.com, huangjun.hj@alibaba-inc.com, yangf@zhejianglab.com, zhangyin98@zju.edu.cn Abstract With the wide availability of Pre-trained Lan- guage Models (PLMs), multi-task fine-tuning across domains has been extensively applied. For tasks related to distant domains with differ- ent class label sets, PLMs may memorize non- transferable knowledge for the target domain and suffer from negative transfer. Inspired by meta-learning, we propose the Meta Distant Transfer Learning (Meta-DTL) framework to learn the cross-task knowledge for PLM-based methods. Meta-DTL first employs task rep- resentation learning to mine implicit relations among multiple tasks and classes. Based on the results, it trains a PLM-based meta-learner to capture the transferable knowledge across tasks. The weighted maximum entropy reg- ularizers are proposed to make meta-learner more task-agnostic and unbiased. Finally, the meta-learner can be fine-tuned to fit each task with better parameter initialization. We evalu- ate Meta-DTL using both BERT and ALBERT on seven public datasets. Experiment results confirm the superiority of Meta-DTL as it con- sistently outperforms strong baselines. We find that Meta-DTL is highly effective when very few data is available for the target task. 1 Introduction Owning to the availability of Pre-trained Language Models (PLMs), the performance of various text classification tasks has been significantly improved. Notable PLMs include BERT (Devlin et al., 2019), ALBERT (Lan et al., 2020), XLNet (Yang et al., 2019), T5 (Raffel et al., 2020), GPT-3 (Brown et al., 2020) and many others. It can be safely concluded that PLM-based approaches achieve state-of-the-art results for a majority of text classification tasks. Among these methods, a key procedure of PLMs is fine-tuning, which enables parameters of PLMs to fit specific datasets. Hence, the performance of PLMs on a downstream task may be limited by ∗ Corresponding author. → {1,2,3,4,5} → {𝑃𝑜𝑠, 𝑁𝑒𝑔} Transfer Multi-task Meta-learner Amazon (Product Reviews) SST-5 (Movie Reviews) Target Target Model (Amazon) Model (SST-5) Fine-tune Meta-learn Fine-tune Figure 1: A simple example of Meta Distant Transfer Learning for review analysis. (Best viewed in color.) the availability of the training set. As reported by several benchmarks such as GLUE (Wang et al., 2019b) and SuperGLUE (Wang et al., 2019a), some PLMs may not perform well in low-resource tasks. A popular solution in NLP is transfer learn- ing (Zhuang et al., 2019; Alyafeai et al., 2020). For PLMs, these models can be fine-tuned over both source-domain and target-domain datasets by var- ious multi-task training strategies (Li et al., 2019; Arase and Tsujii, 2019). Unfortunately, several studies reveal that multi-task training of PLMs across domains does not always guarantee satis- factory results (Sun et al., 2019; Wang et al., 2020). As PLMs usually have large parameter space and strong memorization power, learning from source- domain datasets may force …

Content truncated to 500 words