Frustratingly Simple Pretraining Alternatives to Masked Language Modeling

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3116–3125 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3116 Frustratingly Simple Pretraining Alternatives to Masked Language Modeling Atsuki Yamaguchi1∗, George Chrysostomou2, Katerina Margatina2 and Nikolaos Aletras2 1Research and Development Group, Hitachi, Ltd., Japan 2Department of Computer Science, University of Sheffield, United Kingdom 1atsuki.yamaguchi1@gmail.com 2{gchrysostomou1, k.margatina, n.aletras}@sheffield.ac.uk Abstract Masked language modeling (MLM), a self- supervised pretraining objective, is widely used in natural language processing for learn- ing text representations. MLM trains a model to predict a random sample of input tokens that have been replaced by a [MASK] placeholder in a multi-class setting over the entire vocab- ulary. When pretraining, it is common to use alongside MLM other auxiliary objectives on the token or sequence level to improve down- stream performance (e.g. next sentence predic- tion). However, no previous work so far has attempted in examining whether other simpler linguistically intuitive or not objectives can be used standalone as main pretraining objectives. In this paper, we explore five simple pretrain- ing objectives based on token-level classifica- tion tasks as replacements of MLM. Empirical results on GLUE and SQUAD show that our proposed methods achieve comparable or bet- ter performance to MLM using a BERT-BASE architecture. We further validate our methods using smaller models, showing that pretrain- ing a model with 41% of the BERT-BASE’s pa- rameters, BERT-MEDIUM results in only a 1% drop in GLUE scores with our best objective.1 1 Introduction Masked Language Modeling (MLM) pretrain- ing (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020; Wang et al., 2020) is widely used in natu- ral language processing (NLP) for self-supervised learning of text representations. MLM trains a model (typically a neural network) to predict a par- ticular token that has been replaced with a [MASK] placeholder given its surrounding context. Devlin et al. (2019) first proposed MLM with an additional next sentence prediction (NSP) task (i.e. predicting whether two segments appear consecutively in the original text) to train BERT. ∗Work was done while at the University of Sheffield. 1Our code is publicly available here: https://github. com/gucci-j/light-transformer-emnlp2021 Recently several studies have extended MLM, by masking a contiguous segment of the input in- stead of treating each token independently (Song et al., 2019; Sun et al., 2020; Joshi et al., 2020). Yang et al. (2019) reformulated MLM in XLNET, to mask out attention weights rather than input tokens, such that the input sequence is auto-regressively generated in a random order. ELECTRIC (Clark et al., 2020a) addressed the expensive softmax issue of MLM using a binary classification task, where the task is to distinguish between words sampled from the original data distribution and a noise dis- tribution, using noise-contrastive estimation. In a different direction, previous work has also devel- oped methods to complement MLM for improving text representation learning. Aroca-Ouellette and Rudzicz (2020) have explored sentence and token- level auxiliary pretraining objectives, showing im- provements over NSP. ALBERT (Lan et al., 2020) complemented MLM with …

Content truncated to 500 words