Text Summarization with Pretrained Encoders

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3730–3740, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3730 Text Summarization with Pretrained Encoders Yang Liu and Mirella Lapata Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh yang.liu2@ed.ac.uk, mlap@inf.ed.ac.uk Abstract Bidirectional Encoder Representations from Transformers (BERT; Devlin et al. 2019) rep- resents the latest incarnation of pretrained lan- guage models which have recently advanced a wide range of natural language processing tasks. In this paper, we showcase how BERT can be usefully applied in text summariza- tion and propose a general framework for both extractive and abstractive models. We intro- duce a novel document-level encoder based on BERT which is able to express the semantics of a document and obtain representations for its sentences. Our extractive model is built on top of this encoder by stacking several inter- sentence Transformer layers. For abstractive summarization, we propose a new fine-tuning schedule which adopts different optimizers for the encoder and the decoder as a means of al- leviating the mismatch between the two (the former is pretrained while the latter is not). We also demonstrate that a two-staged fine-tuning approach can further boost the quality of the generated summaries. Experiments on three datasets show that our model achieves state- of-the-art results across the board in both ex- tractive and abstractive settings.1 1 Introduction Language model pretraining has advanced the state of the art in many NLP tasks ranging from sentiment analysis, to question answering, natu- ral language inference, named entity recognition, and textual similarity. State-of-the-art pretrained models include ELMo (Peters et al., 2018), GPT (Radford et al., 2018), and more recently Bidirec- tional Encoder Representations from Transform- ers (BERT; Devlin et al. 2019). BERT combines both word and sentence representations in a single very large Transformer (Vaswani et al., 2017); it is 1Our code is available at https://github.com/ nlpyang/PreSumm. pretrained on vast amounts of text, with an unsu- pervised objective of masked language modeling and next-sentence prediction and can be fine-tuned with various task-specific objectives. In most cases, pretrained language models have been employed as encoders for sentence- and paragraph-level natural language understanding problems (Devlin et al., 2019) involving various classification tasks (e.g., predicting whether any two sentences are in an entailment relationship; or determining the completion of a sentence among four alternative sentences). In this paper, we ex- amine the influence of language model pretrain- ing on text summarization. Different from previ- ous tasks, summarization requires wide-coverage natural language understanding going beyond the meaning of individual words and sentences. The aim is to condense a document into a shorter ver- sion while preserving most of its meaning. Fur- thermore, under abstractive modeling formula- tions, the task requires language generation ca- pabilities in order to create summaries containing novel words and phrases not featured in the source text, while extractive summarization is often de- fined as a binary classification task with …

Content truncated to 500 words