Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6102–6120, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6102 Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation Minki Kang1∗ Moonsu Han1∗ Sung Ju Hwang1,2 KAIST1, Daejeon, South Korea AITRICS2, Seoul, South Korea {zzxc1133, mshan92, sjhwang82}@kaist.ac.kr Abstract We propose a method to automatically gener- ate a domain- and task-adaptive maskings of the given text for self-supervised pre-training, such that we can effectively adapt the language model to a particular target task (e.g. question answering). Specifically, we present a novel re- inforcement learning-based framework which learns the masking policy, such that using the generated masks for further pre-training of the target language model helps improve task per- formance on unseen texts. We use off-policy actor-critic with entropy regularization and experience replay for reinforcement learning, and propose a Transformer-based policy net- work that can consider the relative importance of words in a given text. We validate our Neu- ral Mask Generator (NMG) on several ques- tion answering and text classification datasets using BERT and DistilBERT as the language models, on which it outperforms rule-based masking strategies, by automatically learning optimal adaptive maskings. 1 1 Introduction The recent success of the language model pre- training approaches (Devlin et al., 2019; Peters et al., 2018; Radford et al., 2019; Raffel et al., 2019; Yang et al., 2019), which train language models on diverse text corpora with self-supervised or multi- task learning, have brought up huge performance improvements on several natural language under- standing (NLU) tasks (Wang et al., 2019; Rajpurkar et al., 2016). The key to this success is their ability to learn generalizable text embeddings that achieve near optimal performance on diverse tasks with only a few additional steps of fine-tuning on each downstream task. Most of the existing works on language model aim to obtain a universal language model that can ∗ Equal contribution. 1Code is available at github.com/Nardien/NMG. address nearly the entire set of available natural language tasks on heterogeneous domains. Al- though this train-once and use-anywhere approach has been shown to be helpful for various natu- ral language tasks (Devlin et al., 2019; Radford et al., 2019; Dong et al., 2019; Raffel et al., 2019), there have been considerable needs on adapting the learned language models to domain-specific corpora (e.g. healthcare or legal). Such domains may contain new entities that are not included in the common text corpora, and may contain only a small amount of labeled data as obtaining annotation on them may require expert knowledge. Some recent works (Sun et al., 2019a; Lee et al., 2019; Belt- agy et al., 2019; Gururangan et al., 2020) suggest to further pre-train the language model with self- supervised tasks on the domain-specific text corpus for adaptation, and show that it yields improved performance on tasks from the target domain. Masked Language Models (MLMs) objective in BERT (Devlin et al., 2019) has shown to be effec- tive for the language model to learn the …

Content truncated to 500 words