Denoising based Sequence-to-Sequence Pre-training for Text Generation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4003–4015, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4003 Denoising based Sequence-to-Sequence Pre-training for Text Generation Liang Wang1, Wei Zhao1, Ruoyu Jia1, Sujian Li2, Jingming Liu1 1Yuanfudao AI Lab, Beijing, China 2Key Laboratory of Computational Linguistics, Peking University, MOE, China {wangliang01,zhaowei01,jiary,liujm}@fenbi.com lisujian@pku.edu.cn Abstract This paper presents a new sequence-to- sequence (seq2seq) pre-training method PoDA (Pre-training of Denoising Autoencoders), which learns representations suitable for text generation tasks. Unlike encoder-only (e.g., BERT) or decoder-only (e.g., OpenAI GPT) pre-training approaches, PoDA jointly pre- trains both the encoder and decoder by denois- ing the noise-corrupted text, and it also has the advantage of keeping the network architecture unchanged in the subsequent fine-tuning stage. Meanwhile, we design a hybrid model of Transformer and pointer-generator networks as the backbone architecture for PoDA. We conduct experiments on two text generation tasks: abstractive summarization, and gram- matical error correction. Results on four datasets show that PoDA can improve model performance over strong baselines without us- ing any task-specific techniques and signifi- cantly speed up convergence. 1 1 Introduction Methods based on unsupervised pre-training and supervised fine-tuning for NLP have achieved phe- nomenal successes in the last two years. Most of the proposed methods in the literature choose lan- guage modeling or its variant as the pre-training task. After the pre-training stage, ELMo (Pe- ters et al., 2018) and CoVe (McCann et al., 2017) directly use the learned representations as additional features for downstream tasks, while BERT (Devlin et al., 2018), ULMFiT (Howard and Ruder, 2018), XLM (Lample and Conneau, 2019), and OpenAI GPT (Radford et al., 2018, 2019) require fine-tuning both pre-trained param- eters and task-specific parameters on labeled data. The state-of-the-art performances have been sig- nificantly advanced for classification and sequence 1The code and pre-trained models are available at https: //github.com/yuantiku/PoDA. labeling tasks, such as natural language inference (Bowman et al., 2015), named-entity recognition, SQuAD question answering (Rajpurkar et al., 2016) etc. However, little attention has been paid to pre- training for seq2seq text generation (Sutskever et al., 2014). A typical seq2seq network con- sists of a bidirectional encoder, a unidirectional decoder and attention between the encoder and de- coder. Previous work mainly focuses on encoder- only or decoder-only pre-training. For example, BERT pre-trains a bidirectional encoder, and Ope- nAI GPT pre-trains a language model which is es- sentially a unidirectional decoder. Ramachandran et al. (2016) propose to train two independent lan- guage models for the encoder and decoder respec- tively. All of the aforementioned methods are only able to partially pre-train the seq2seq networks, and therefore are unable to unleash the full poten- tial of transfer learning for text generation. In this paper, we present PoDA, a denoising based pre-training method that is able to jointly pre-train all components of seq2seq networks. Like denoising autoencoders, PoDA works by de- noising the noise-corrupted text sequences. Any noising function that fits …

Content truncated to 500 words