SCRIPT: Self-Critic PreTraining of Transformers

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5196–5202 June 6–11, 2021. ©2021 Association for Computational Linguistics 5196 SCRIPT: Self-Critic Pretraining of Transformers Erik Nijkamp∗ UCLA enijkamp@ucla.edu Bo Pang UCLA bopang@ucla.edu Ying Nian Wu UCLA ywu@stat.ucla.edu Caiming Xiong Salesforce Research cxiong@salesforce.com Abstract We introduce Self-CRItic Pretraining Trans- formers (SCRIPT) for representation learning of text. The popular masked language mod- eling (MLM) pretraining methods like BERT replace some tokens with [MASK] and an en- coder is trained to recover them, while ELEC- TRA trains a discriminator to detect replaced tokens proposed by a generator. In contrast, we train a language model as in MLM and fur- ther derive a discriminator or critic on top of the encoder without using any additional pa- rameters. That is, the model itself is a critic. SCRIPT combines MLM training and dis- criminative training for learning rich represen- tations and compute- and sample-efficiency. We demonstrate improved sample-efficiency in pretraining and enhanced representations ev- idenced by improved downstream task perfor- mance on GLUE and SQuAD over strong base- lines. Also, the self-critic scores can be di- rectly used as pseudo-log-likelihood for effi- cient scoring. 1 Introduction In natural language processing, the landscape of unsupervised learning methods is dominated by masked language modeling (MLM) for bi- directional encoders, such as BERT (Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019; Joshi et al., 2020; Lan et al., 2019; Lewis et al., 2020; Jiao et al., 2019), and causal masking for uni-directional auto- regressive decoders (Radford et al., 2018, 2019; Brown et al., 2020; Raffel et al., 2020; Lewis et al., 2019) such as GPT. In MLM an encoder is pre- trained on a generic corpus of text with the hope of learning universal contextual embeddings, which, then, are fine-tuned on a specific down-stream task. Whereas recent developments in causal masking aim to learn a large-scale model once and define the down-stream task as an auto-regressive man- ner in the form of few-shot evaluation (Brown ∗ Research conducted at Salesforce Einstein et al., 2020). In practice, while an universal auto- regressive neural backbone model without the need for fine-tuning such as GPT-3 is desirable, the com- putational complexity at inference time remains an open problem. While the two-stage approach of MLM of smaller models is computationally conve- nient, the pretraining still incurs a substantial com- putational cost. Hence, in this work, we focus on learning contextual bi-directional representations with the goal of improving upon sample efficiency. In MLM, the input sequence of tokens is per- turbed by randomly masking out a small subset of the identities of tokens (Devlin et al., 2018) or attention scores to those tokens (Yang et al., 2019). Then, the generative model is learned as a denois- ing auto-encoder (Vincent et al., 2008) which re- covers the masked out tokens. While the learned contextual representations achieve remarkable per- formance on down-stream tasks, the pretraining requires substantial compute. This is mainly …

Content truncated to 500 words