Abstractive Text Summarization based on Language Model Conditioning and Locality Modeling

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6680–6689 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6680 Abstractive Text Summarization based on Language Model Conditioning and Locality Modeling Dmitrii Aksenov, Juli´an Moreno-Schneider, Peter Bourgonje, Robert Schwarzenberg, Leonhard Hennig, Georg Rehm DFKI GmbH, Alt-Moabit 91c, 10559 Berlin, Germany {firstname.lastname}@dfki.de Abstract We explore to what extent knowledge about the pre-trained language model that is used is beneficial for the task of abstractive summarization. To this end, we experiment with conditioning the encoder and decoder of a Transformer-based neural model on the BERT language model. In addition, we propose a new method of BERT-windowing, which allows chunk-wise processing of texts longer than the BERT window size. We also explore how locality modeling, i. e., the explicit restriction of calculations to the local context, can affect the summarization ability of the Transformer. This is done by introducing 2-dimensional convolutional self-attention into the first layers of the encoder. The results of our models are compared to a baseline and the state-of-the-art models on the CNN/Daily Mail dataset. We additionally train our model on the SwissText dataset to demonstrate usability on German. Both models outperform the baseline in ROUGE scores on two datasets and show its superiority in a manual qualitative analysis. Keywords: Summarisation, Language Modeling, Information Extraction, Information Retrieval, BERT, Locality Modeling 1. Introduction Text summarization is an NLP task with many real-world applications. The ever-increasing amount of unstructured information in text form calls for methods to automati- cally extract the relevant information from documents and present it in condensed form. Within the field of sum- marization, different paradigms are recognised in two di- mensions: extractive vs. abstractive, and single-document vs. multi-document. In extractive summarization, those sentences or words are extracted from a text which carry the most important information, directly presenting the re- sult of this as the summary. Abstractive summarization methods paraphrase the text, and by changing the text aim to generate more flexible and consistent summaries. Fur- thermore, single-document summarization works on sin- gle documents, while multi-document summarization deals with multiple documents at once and produces a single summary. In this paper, we concentrate on single-document abstractive summarization. Most recent abstractive models utilize the neural network-based sequence-to-sequence ap- proach. During training, such models calculate the condi- tional probability of a summary given the input sequence by maximizing the loss function (typically cross-entropy). Most approaches are based on the encoder-decoder frame- work where the encoder encodes the input sequence into a vector representation and the decoder produces a new sum- mary given the draft summary (which is the part of the summary generated during previous iterations). The last layer of a decoder, the generator, maps hidden states to to- ken probabilities. We use a state-of-the-art Transformer for sequence-to-sequence tasks which is built primarily on the attention mechanism (Vaswani et al., 2017). We attempt to improve performance of abstractive text sum- marization by improving the language encoding capabili- ties of the model. Recent …

Content truncated to 500 words