Train No Evil: Selective Masking for Task-Guided Pre-Training

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6966–6974, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6966 Train No Evil: Selective Masking for Task-Guided Pre-Training Yuxian Gu1,2,3, Zhengyan Zhang1,2,3, Xiaozhi Wang1,2,3, Zhiyuan Liu1,2,3†, Maosong Sun1,2,3 1Department of Computer Science and Technology, Tsinghua University, Beijing, China 2Institute for Artificial Intelligence, Tsinghua University, Beijing, China 3State Key Lab on Intelligent Technology and Systems, Tsinghua University, Beijing, China {gu-yx17, zy-z19, wangxz20}@mails.tsinghua.edu.cn Abstract Recently, pre-trained language models mostly follow the pre-train-then-fine-tuning paradigm and have achieved great performance on vari- ous downstream tasks. However, since the pre- training stage is typically task-agnostic and the fine-tuning stage usually suffers from in- sufficient supervised data, the models cannot always well capture the domain-specific and task-specific patterns. In this paper, we pro- pose a three-stage framework by adding a task-guided pre-training stage with selective masking between general pre-training and fine- tuning. In this stage, the model is trained by masked language modeling on in-domain unsupervised data to learn domain-specific patterns and we propose a novel selective masking strategy to learn task-specific pat- terns. Specifically, we design a method to measure the importance of each token in se- quences and selectively mask the important tokens. Experimental results on two senti- ment analysis tasks show that our method can achieve comparable or even better per- formance with less than 50% of computation cost, which indicates our method is both effec- tive and efficient. The source code of this pa- per can be obtained from https://github. com/thunlp/SelectiveMasking. 1 Introduction Pre-trained Language Models (PLMs) have achieved superior performances on various NLP tasks (Baevski et al., 2019; Joshi et al., 2020; Liu et al., 2019; Yang et al., 2019; Clark et al., 2020) and have attracted wide research interests. Inspired by the success of GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), most PLMs follow the pre-train-then-fine-tuning paradigm, which adopts unsupervised pre-training on large general-domain † Corresponding author: Z.Liu (liuzy@tsinghua.edu.cn) General Pre-train In-domain Unsupervised Data General Unsupervised Data Downstream Supervised Data Fine-tune Selective Masking Random Masking Task-guided Pre-train Figure 1: The overall three-stage framework. We add task-guided pre-training between general pre-training and fine-tuning to efficiently and effectively learn the domain-specific and task-specific language patterns. corpora to learn general language patterns and su- pervised fine-tuning to adapt to downstream tasks. Recently, Gururangan et al. (2020) shows that learning domain-specific and task-specific patterns during pre-training can be helpful to the models for certain domains and tasks. However, conven- tional pre-training is aimless with respect to spe- cific downstream tasks, and fine-tuning usually suf- fers from insufficient supervised data, preventing PLMs from effectively capturing these patterns. To learn domain-specific language patterns, some previous works (Beltagy et al., 2019; Huang et al., 2020) pre-train a BERT-like model from scratch using large-scale in-domain data. However, they are computation-intensive and require large- scale in-domain data, which is hard to obtain in many domains. To learn task-specific language pat- terns, some previous works (Phang et al., 2018) add intermediate supervised pre-training …

Content truncated to 500 words