GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 9394–9412 July 9-14, 2023 ©2023 Association for Computational Linguistics GANLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator Jian Yang1 ∗, Shuming Ma2, Li Dong2, Shaohan Huang2, Haoyang Huang2, Yuwei Yin3, Dongdong Zhang2, Liqun Yang1 †, Furu Wei2, Zhoujun Li1 1State Key Lab of Software Development Environment, Beihang University 2Microsoft Research Asia; 3The University of Hong Kong {jiaya, lqyang, lizj}@buaa.edu.cn; {shumma, lidong1, shaohanh, haohua, dozhang, fuwei}@microsoft.com; yuweiyin@hku.hk Abstract Pre-trained models have achieved remarkable success in natural language processing (NLP). However, existing pre-training methods under- utilize the benefits of language understanding for generation. Inspired by the idea of Genera- tive Adversarial Networks (GANs), we propose a GAN-style model for encoder-decoder pre- training by introducing an auxiliary discrimi- nator, unifying the ability of language under- standing and generation in a single model. Our model, named as GANLM, is trained with two pre-training objectives: replaced token detec- tion and replaced token denoising. Specifically, given masked source sentences, the generator outputs the target distribution and the discrim- inator predicts whether the target sampled to- kens from distribution are incorrect. The target sentence is replaced with misclassified tokens to construct noisy previous context, which is used to generate the gold sentence. In gen- eral, both tasks improve the ability of lan- guage understanding and generation by selec- tively using the denoising data. Extensive ex- periments in language generation benchmarks show that GANLM with the powerful language understanding capability outperforms various strong pre-trained language models (PLMs) and achieves state-of-the-art performance.1 1 Introduction The pre-training-then-fine-tuning paradigm has been proven successful in many natural language processing tasks (Devlin et al., 2019; Liu et al., 2019; Schick and Schütze, 2021). While there are various pre-training approaches for the encoder- only architectures (Clark et al., 2020; Conneau et al., 2020), the encoder-decoder pre-training is un- derexplored, which is essential for natural language generation. To pre-train the entire encoder-decoder Asia.∗Contribution during internship at Microsoft Research †Corresponding author. 1https://github.com/CSJianYang/GanLM The [MASK] the flowers Generator Input guardian watered Generator Prediction (a) Replaced Token Detection (b) Replaced Token Denoising Discriminator Target Discriminator Prediction The gardener watered the flowers Original Sentence [BOS] guardian watering Generator Input gardener watered [EOS] Generator Target ORIGINAL REPLACED REPLACED ORIGINAL Figure 1: A pre-training sample of our method, where replaced token detection (discriminator) and replaced token denoising (generator) are used for pre-training. The discriminator classifies each generated token into REPLACED or ORIGINAL, where REPLACED denotes the predicted token is different from the gold token. The red fonts denote incorrect predictions. model, BART (Lewis et al., 2020) proposes a de- noising language model objective and T5 (Raffel et al., 2020) pre-trains the models with a span cor- ruption objective. Furthermore, mBART (Liu et al., 2020) and mT5 (Xue et al., 2021) extend them to be multilingual pre-trained language models. Unlike most encoder-decoder pre-training meth- ods that simply apply sequence-to-sequence tasks on a single encoder-decoder architecture, we ex- plore the approaches to pre-train the model in a …

Content truncated to 500 words