Meta Fine-Tuning Neural Language Models for Multi-Domain Text Mining

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3094–3104, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3094 Meta Fine-Tuning Neural Language Models for Multi-Domain Text Mining Chengyu Wang1, Minghui Qiu1∗, Jun Huang1, Xiaofeng He2 1 Alibaba Group 2 East China Normal University {chengyu.wcy, minghui.qmh, huangjun.hj}@alibaba-inc.com hexf@cs.ecnu.edu.cn Abstract Pre-trained neural language models bring sig- nificant improvement for various NLP tasks, by fine-tuning the models on task-specific training sets. During fine-tuning, the param- eters are initialized from pre-trained models directly, which ignores how the learning pro- cess of similar NLP tasks in different domains is correlated and mutually reinforced. In this paper, we propose an effective learning proce- dure named Meta Fine-Tuning (MFT), serving as a meta-learner to solve a group of similar NLP tasks for neural language models. In- stead of simply multi-task training over all the datasets, MFT only learns from typical in- stances of various domains to acquire highly transferable knowledge. It further encour- ages the language model to encode domain- invariant representations by optimizing a se- ries of novel domain corruption loss functions. After MFT, the model can be fine-tuned for each domain with better parameter initializa- tion and higher generalization ability. We im- plement MFT upon BERT to solve several multi-domain text mining tasks. Experimental results confirm the effectiveness of MFT and its usefulness for few-shot learning. 1 1 Introduction Recent years has witnessed a boom in pre-trained neural language models. Notable works include ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), Transformer-XL (Dai et al., 2019), AL- BERT (Lan et al., 2019), StructBERT (Wang et al., 2019b) and many others. These models revolution- ize the learning paradigms of various NLP tasks. After pre-training, only a few fine-tuning epochs are required to train models for these tasks. The “secrets” behind this phenomenon owe to the models’ strong representation learning power ∗Corresponding author. 1Our code will be available at: https://github. com/alibaba/EasyTransfer/. to encode the semantics and linguistic knowledge from massive text corpora (Jawahar et al., 2019; Ko- valeva et al., 2019; Liu et al., 2019a; Tenney et al., 2019). By simple fine-tuning, models can trans- fer the universal Natural Language Understand- ing (NLU) abilities to specific tasks (Wang et al., 2019a). However, state-of-the art language mod- els mostly utilize self-supervised tasks during pre- training (for instance, masked language modeling and next sentence prediction in BERT (Devlin et al., 2019)). This unavoidably creates a learning gap between pre-training and fine-tuning. Besides, for a group of similar tasks, conventional practices re- quire the parameters of all task-specific models to be initialized from the same pre-trained language model, ignoring how the learning process in differ- ent domains is correlated and mutually reinforced. A basic solution is fine-tuning models by multi- task learning. Unfortunately, multi-task fine-tuning of BERT does not necessarily yield better perfor- mance across all the tasks (Sun et al., 2019a). A probable cause is that learning too much from other tasks may force the model to acquire non- transferable knowledge, which …

Content truncated to 500 words