A Summarization Dataset of Slovak News Articles

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6725–6730 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6725 A Summarization Dataset of Slovak News Articles Marek ˇSuppa, Jerguˇs Adamec Faculty of Mathematics, Physics and Informatics Comenius University in Bratislava, Slovakia marek@suppa.sk, jergus.adamec@gmail.com Abstract As a well established NLP task, single-document summarization has seen significant interest in the past few years. However, most of the work has been done on English datasets. This is particularly noticeable in the context of evaluation where the dominant ROUGE metric assumes its input to be written in English. In this paper we aim to address both of these issues by introducing a summarization dataset of articles from a popular Slovak news site and proposing small adaptation to the ROUGE metric that make it better suited for Slovak texts. Several baselines are evaluated on the dataset, including an extractive approach based on the Multilingual version of the BERT architecture. To the best of our knowledge, the presented dataset is the first large-scale news-based summarization dataset for text written in Slovak language. It can be reproduced using the utilities available at https://github.com/NaiveNeuron/sme-sum. Keywords: summarization dataset, document summarization, neural networks, Slovak 1. Introduction Document summarization is a well-estabilished NLP task which has received considerable attention in the past few years, especially its single-document summarization vari- ant. It is considered to be particularly challenging, as the automatic system needs to understand the text well enough to identify its most important parts, so that it can produce the final summary by either aggregating or rephrasing the previously identified content. Depending on whether they lean towards the former or the latter, such systems can be characterized as extractive or abstractive (Gambhir and Gupta, 2017). Much of the recent progress in this area can be attributed to the use of models based on neural networks. While they have shown promising results, particularly in the area of abstractive summarization, they require training corpora on the order of thousands of documents. This has led the au- thors of (Dernoncourt et al., 2018) to conclude that more large-scale corpora for summarization than those already available are needed. The issue is much more pronounced in the context of summarization of non-English texts, as the data there is small or virtually non-existent. Further- more, the progress in this area is hindered by the fact that the evaluation metrics used to compare document summa- rization systems are English-centric (i.e. they depend on English stemmers and stop-words) and cannot therefore be directly applied in other languages. In this work we try to address both of these issues by in- troducing a document summarization dataset which con- sists of Slovak news stories obtained from a prominent Slo- vak news website. To evaluate its performance, we use a slightly altered version of the ROUGE metric (Lin and Hovy, 2003) which aims to provide a more realistic com- parison by utilizing a Slovak stemmer. 2. Related Work As a task, text summarization …

Content truncated to 500 words