Rethinking Denoised Auto-Encoding in Language Pre-Training

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2922–2932 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2922 Rethinking Denoised Auto-Encoding in Language Pre-Training Fuli Luo1∗, Pengcheng Yang1,2∗, Shicheng Li2, Xuancheng Ren2, Xu Sun2, Songfang Huang1, Fei Huang1 1Alibaba Group 2Key Laboratory of Computational Linguistics, Peking University {lfl259702,songfang.hsf,f.huang}@alibaba-inc.com {yang_pc,lisc99,renxc,xusun}@pku.edu.cn Abstract Pre-trained self-supervised models such as BERT have achieved striking success in learn- ing sequence representations, especially for natural language processing. These models typically corrupt the given sequences with cer- tain types of noise, such as masking, shuffling, or substitution, and then try to recover the original input. However, such pre-training ap- proaches are prone to learning representations that are covariant with the noise, leading to the discrepancy between the pre-training and fine- tuning stage. To remedy this, we present Con- trAstive Pre-Training (CAPT) to learn noise invariant sequence representations. The pro- posed CAPT encourages the consistency be- tween representations of the original sequence and its corrupted version via unsupervised instance-wise training signals. In this way, it not only alleviates the pretrain-finetune dis- crepancy induced by the noise of pre-training, but also aids the pre-trained model in better capturing global semantics of the input via more effective sentence-level supervision. Dif- ferent from most prior work that focuses on a particular modality, comprehensive empir- ical evidence on 11 natural language under- standing and cross-modal tasks illustrates that CAPT is applicable for both language and vision-language tasks, and obtains surprisingly consistent improvement, including 0.6% abso- lute gain on GLUE benchmarks and 0.8% ab- solute increment on NLVR2. 1 Introduction Recently, pre-trained self-supervised models such as BERT (Devlin et al., 2019) have attracted an increasing amount of attention in natural lan- guage processing and vision-language process- ing. Benefiting from common knowledge con- tained in massive unlabeled data (Liu et al., 2019), the pretraining-finetuning framework has become ∗Equal Contribution. Models Noise types BERT (Devlin et al., 2019) Mask tokens SpanBERT (Joshi et al., 2019) Mask spans RoBERTa (Liu et al., 2019) Mask token XLNet (Yang et al., 2019) Shuffle token ELECTRA (Clark et al., 2019) Replace tokens StructBERT (Wang et al., 2019b) Mask + Shuffle tokens BART (Lewis et al., 2019) Mask + Shuffle + Replace. UNITER (Chen et al., 2019) Mask tokens/regions LXMERT (Tan and Bansal, 2019) Mask tokens/regions Table 1: The type of noise used in the current natu- ral language (upper) and vision-language (lower) se- quence representation models. a representative paradigm for advancing various language-related downstream tasks. Most endeavors on pre-trained representa- tion models rely on elaborately designed self- supervised tasks, which typically corrupt the given sequence with certain types of noise (e.g., masking in BERT in Table 1), and then train the model to re- cover the original sequence. As a consequence, the learned representations tend to be covariant with the input noise of pre-training in this paradigm. However, when transferred to downstream tasks, the pre-trained model is responsible for encoding the original sequence without noise, and is ex- pected to obtain noise invariant representations. Such …

Content truncated to 500 words