PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8681–8691, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8681 PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation Bin Bi, Chenliang Li, Chen Wu, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, Luo Si Alibaba Group {b.bi, lcl193798, wuchen.wc, ym119608}@alibaba-inc.com {hebian.ww, songfang.hsf, f.huang, luo.si}@alibaba-inc.com Abstract Self-supervised pre-training, such as BERT (Devlin et al., 2018), MASS (Song et al., 2019) and BART (Lewis et al., 2019), has emerged as a powerful technique for nat- ural language understanding and generation. Existing pre-training techniques employ au- toencoding and/or autoregressive objectives to train Transformer-based models by recovering original word tokens from corrupted text with some masked tokens. The training goals of existing techniques are often inconsistent with the goals of many language generation tasks, such as generative question answering and conversational response generation, for producing new text given context. This work presents PALM with a novel scheme that jointly pre-trains an autoencod- ing and autoregressive language model on a large unlabeled corpus, specifically designed for generating new text conditioned on con- text. The new scheme alleviates the mismatch introduced by the existing denoising scheme between pre-training and fine-tuning where generation is more than reconstructing orig- inal text. An extensive set of experiments show that PALM achieves new state-of-the- art results on a variety of language genera- tion benchmarks covering generative question answering (Rank 1 on the official MARCO leaderboard), abstractive summarization on CNN/DailyMail as well as Gigaword, ques- tion generation on SQuAD, and conversational response generation on Cornell Movie Dia- logues. 1 Introduction Self-supervised pre-training has achieved great suc- cess in a wide range of natural language under- standing (NLU) tasks (Dai and Le, 2015; Howard and Ruder, 2018; Radford, 2018; Peters et al., 2018; Devlin et al., 2018). Different from lan- guage understanding, language generation aims at generating natural language sentences, including tasks like neural machine translation (Bahdanau et al., 2015; Vaswani et al., 2017), abstractive sum- marization (Rush et al., 2015; See et al., 2017a; Gehrmann et al., 2018), generative question an- swering (QA) (Tan et al., 2017; Bi et al., 2019), question generation (Zhao et al., 2018) and con- versational response generation (Vinyals and Le, 2015). Many of the language generation tasks re- quire the models to read and to comprehend a given document, based on which output text is generated. In this paper, we present PALM, a novel approach to Pre-training an Autoencoding&autoregressive Language Model for text generation based on read- ing comprehension of textual context. Recently, several pre-training methods have been proposed for language generation. GPT (Radford, 2018) and GPT-2 (Radford et al., 2019) use a left- to-right Transformer decoder to generate a text se- quence token-by-token, which lacks an encoder to condition generation on context. In contrast, MASS (Song et al., 2019) and BART (Lewis et al., 2019) both employ a Transformer-based encoder- decoder framework, with a bidirectional encoder over corrupted (masked) text and a left-to-right decoder reconstructing the original text. While such denoising …

Content truncated to 500 words