Generating Extended and Multilingual Summaries with Pre-trained Transformers

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1640–1650 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1640 Generating Extended and Multilingual Summaries with Pre-trained Transformers R´emi Calizzano, Malte Ostendorff, Qian Ruan, Georg Rehm DFKI GmbH, Alt-Moabit 91c, 10559 Berlin, Germany {remi.calizzano, malte.ostendorff, qian.ruan, georg.rehm}@dfki.de Abstract Almost all summarisation methods and datasets focus on a single language and short summaries. We introduce a new dataset called WikinewsSum for English, German, French, Spanish, Portuguese, Polish, and Italian summarisation tailored for extended summaries of approx. 11 sentences. The dataset comprises 39,626 summaries which are news articles from Wikinews and their sources. We compare three multilingual transformer models for extractive summarisation and three training scenarios on which we fine-tune mT5 to perform abstractive summarisation. This results in strong baselines for both extractive and abstractive summarisation on WikinewsSum. We also show how the combination of an extractive model with an abstractive one can be used to create extended abstractive summaries from long input documents. Our results show that fine-tuning mT5 on all the languages combined significantly improves the summarisation performance on low-resource languages. Keywords: Summarisation, Multilingualism, Extended summarisation, Dataset 1. Introduction Summarisation is a well-known Natural Language Pro- cessing (NLP) task. Recently, the quality of automati- cally generated summaries increased with the appear- ance of the transformer architecture (Vaswani et al., 2017) and the development of pre-trained sentence-to- sentence models such as BART (Lewis et al., 2019), T5 (Raffel et al., 2019), and Pegasus (Zhang et al., 2020a). These models improve the state-of-the-art in various tasks including summarisation on the CNN/Daily Mail (CNN-DM) dataset (Hermann et al., 2015), for exam- ple. However, different types of input content, and dif- ferent desired types and lengths of the output summary, call for different methods altogether, or different pa- rameter settings when using the same method. This paper focuses on generating relatively long sum- maries for multiple languages. In contrast to popular datasets for English summarisation (Hermann et al., 2015; Over and Liggett, 2002; Narayan et al., 2018; Graff et al., 2003; Grusky et al., 2018) or multilin- gual summarisation (Scialom et al., 2020; Hasan et al., 2021; Ladhak et al., 2020), in which the average length of summaries ranges between one and five sentences (Dernoncourt et al., 2018), we aim to generate longer summaries of around 11 sentences. Throughout our pa- per, we will use the term extended summaries to refer to summaries with a length of approximately eleven sen- tences. Thus, we call the task multilingual, extended text summarisation. Only a few summarisation datasets are tailored to generate summaries of this length. Con- sidering our task and focus, Multi-News (Fabbri et al., 2019) and WikiSum (Liu et al., 2018) are the most suit- able datasets. However, to the best of our knowledge, there is currently no public non-English dataset that is suitable for our purpose. To fill this gap, we make use of Wikinews to create the first multilingual summarisa- tion dataset tailored to extended …

Content truncated to 500 words