NB-MLM: Efficient Domain Adaptation of Masked Language Models for Sentiment Analysis

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9114–9124 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9114 NB-MLM: Efficient Domain Adaptation of Masked Language Models for Sentiment Analysis Nikolay Arefyev♦,∇,△ Dmitrii Kharchev♦,∇ Artem Shelmanov⃝,□,∇ ♦Samsung Research Center Russia / Moscow, Russia ∇Lomonosov Moscow State University / Moscow, Russia △National Research University Higher School of Economics / Moscow, Russia ⃝Artificial Intelligence Research Institute / Moscow, Russia □Sber AI Lab / Moscow, Russia {nick.arefyev,dimitriy.kharchev}@gmail.com shelmanov@airi.net Abstract While Masked Language Models (MLM) are pre-trained on massive datasets, the additional training with the MLM objective on domain or task-specific data before fine-tuning for the final task is known to improve the final perfor- mance. This is usually referred to as the do- main or task adaptation step. However, unlike the initial pre-training, this step is performed for each domain or task individually and is still rather slow, requiring several GPU days com- pared to several GPU hours required for the final task fine-tuning. We argue that the standard MLM objective leads to inefficiency when it is used for the adaptation step because it mostly learns to pre- dict the most frequent words, which are not necessarily related to a final task. We pro- pose a technique for more efficient adaptation that focuses on predicting words with large weights of the Naive Bayes classifier trained for the task at hand, which are likely more rel- evant than the most frequent words. The pro- posed method provides faster adaptation and better final performance for sentiment analysis compared to the standard approach. 1 Introduction Pre-training of neural networks with a language model (LM) or masked language model (MLM) objective on large amounts of non-domain-specific texts has given a significant boost of performance in almost all natural language processing tasks. While 16GB of texts were shown to BERT (Devlin et al., 2019) and ten times more to RoBERTa (Liu et al., 2019) during pre-training, the further training of these models with the MLM objective on domain- specific texts before fine-tuning to the target task was shown to further improve the final results (Sun et al., 2019; Gururangan et al., 2020). This tech- nique is called the domain or task adaptation, de- pending on the degree of similarity of the data for adaptation to the target dataset. While initial pre- training is extremely expensive, it does not depend on the final task and can be performed only once. However, domain or task adaptation is done for each domain or task individually and is still quite resource-demanding, requiring hundreds of thou- sands of training steps or several GPU days, unlike final fine-tuning, which can often be done in a few GPU hours (Sun et al., 2019). In this work, we propose a method for more ef- ficient MLM adaptation. We have noticed that the standard MLM spends most of the training time on learning to restore the most frequent words like de- terminers or auxiliary verbs hidden (masked) from its input. While such training …

Content truncated to 500 words