Pretraining-Based Natural Language Generation for Text Summarization

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 789–797 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 789 Pretraining-Based Natural Language Generation for Text Summarization Haoyu Zhang†, Jingjing Cai‡, Jianjun Xu†, Ji Wang† †HPCL, College of Computer, National University of Defense Technology, China ‡Dalian Neusoft University of Information, China {zhanghaoyu10, jjxu, wj}@nudt.edu.cn caijingjing@neusoft.edu.cn Abstract In this paper, we propose a novel pretraining- based encoder-decoder framework, which can generate the output sequence based on the in- put sequence in a two-stage manner. For the encoder of our model, we encode the input sequence into context representations using BERT. For the decoder, there are two stages in our model, in the first stage, we use a Transformer-based decoder to generate a draft output sequence. In the second stage, we mask each word of the draft sequence and feed it to BERT, then by combining the input se- quence and the draft representation generated by BERT, we use a Transformer-based decoder to predict the refined word for each masked position. To the best of our knowledge, our approach is the first method which applies the BERT into text generation tasks. As the first step in this direction, we evaluate our proposed method on the text summarization task. Exper- imental results show that our model achieves new state-of-the-art on both CNN/Daily Mail and New York Times datasets. 1 Introduction Text summarization generates summaries from in- put documents while keeping salient information. It is an important task and can be applied to sev- eral real-world applications. Many methods have been proposed to solve the text summarization problem (See et al., 2017; Nallapati et al., 2017; Zhou et al., 2018; Gehrmann et al., 2018). There are two main text summarization techniques: ex- tractive and abstractive. Extractive summarization generates summary by selecting salient sentences or phrases from the source text, while abstractive methods paraphrase and restructure sentences to compose the summary. We focus on abstractive summarization in this work as it is more flexible and thus can generate more diverse summaries. Recently, many abstractive approaches are in- troduced based on neural sequence-to-sequence framework (Paulus et al., 2018; See et al., 2017; Gehrmann et al., 2018; Li et al., 2018). Based on the sequence-to-sequence model with copy mech- anism, (See et al., 2017) incorporates a cover- age vector to track and control attention scores on source text. (Paulus et al., 2018) introduce intra- temporal attention processes in the encoder and decoder to address the repetition and incoherent problem. There are two issues in previous abstractive methods: 1) these methods use left-context-only decoder, thus do not have complete context when predicting each word. 2) they do not utilize the pre-trained contextualized language models on the decoder side, so it is more difficult for the decoder to learn summary representations, context interac- tions and language modeling together. Recently, BERT has been successfully used in various natural language processing tasks, such as textual entailment, name entity recognition and machine reading comprehensions. In this paper, we …

Content truncated to 500 words