Instance Regularization for Discriminative Language Model Pre-training

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11255 - 11265 December 7-11, 2022 ©2022 Association for Computational Linguistics Instance Regularization for Discriminative Language Model Pre-training Zhuosheng Zhang1,2*, Hai Zhao1,2, Ming Zhou3 1Department of Computer Science and Engineering, Shanghai Jiao Tong University 2Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University 3Langboat Technology zhangzs@sjtu.edu.cn, zhaohai@cs.sjtu.edu.cn, zhouming@chuangxin.com Abstract Discriminative pre-trained language models (PrLMs) can be generalized as denoising auto-encoders that work with two procedures, ennoising and denoising. First, an ennoising process corrupts texts with arbitrary noising functions to construct training instances. Then, a denoising language model is trained to restore the corrupted tokens. Existing studies have made progress by optimizing independent strategies of either ennoising or denosing. They treat training instances equally throughout the training process, with little attention on the individual contribution of those instances. To model explicit signals of instance contribution, this work proposes to estimate the complexity of restoring the original sentences from corrupted ones in language model pre- training. The estimations involve the corruption degree in the ennoising data construction process and the prediction confidence in the denoising counterpart. Experimental results on natural language understanding and reading comprehension benchmarks show that our approach improves pre-training efficiency, effectiveness, and robustness. Code is publicly available at https://github.com/cooelf/ InstanceReg 1 Introduction Leveraging self-supervised objectives to pre-train language models (PrLMs) on massive unlabeled data has shown success in natural language processing (NLP) (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019; Dong et al., 2019; Lan et al., 2020; Clark et al., 2020; Luo et al., 2021; Zhu et al., 2022). A wide landscape of pre-training objectives has been produced, such as autoregressive (Radford et al., 2018; Yang et al., 2019) and autoencoding (Devlin et al., 2019; * Work done during internship at Lanboat. This work was partially supported by Key Projects of National Natural Science Foundation of China (U1836222 and 61733011). Ennoising Denoising [MASK] cute dog [MASK] playing on the [MASK] ... A cute [MASK] is [MASK] on the [MASK] ... Original Sentence Corrupted Sentence Restored Sentence Discriminative Training Figure 1: Overview of AutoDecoders. As the two examples show, the random sampling operation during ennoising would result in training instances of different degree of difficulty, e.g., the variety of valid alternatives. Joshi et al., 2020) language modeling objectives, which serve as the principled mechanisms to teach language models general-purpose knowledge through the pre-training, and then those pre- trained PrLMs can be fine-tuned for downstream tasks. Based on these unsupervised functions, three classes of PrLMs have been proposed: autoregressive language models (e.g. GPT (Radford et al., 2018)), autoencoding models (e.g. BERT (Devlin et al., 2019)), and encoder- decoder models (e.g. BART (Lewis et al., 2020a) and T5 (Raffel et al., 2020)). In this work, we focus on the research line of autoencoding models, also known as discriminative PrLMs that have achieved impressive performance on natural language understanding (NLU). Although the discriminative PrLMs may vary in language …

Content truncated to 500 words