SumeCzech: Large Czech News-Based Summarization Dataset

SumeCzech: Large Czech News-Based Summarization Dataset Milan Straka, Nikita Mediankin, Tom Kocmi, Zdenˇek ˇZabokrtsk´y, Vojtˇech Hudeˇcek, Jan Hajiˇc Institute of Formal and Applied Linguistics Charles University, Faculty of Mathematics and Physics Malostransk´e n´amˇest´ı 25, Prague, Czech Republic {straka,mediankin,kocmi,zabokrtsky,hudecek,hajic}@ufal.mff.cuni.cz Abstract Document summarization is a well-studied NLP task. With the emergence of artificial neural network models, the summarization performance is increasing, as are the requirements on training data. However, only a few datasets are available for Czech, none of them particularly large. Additionally, summarization has been evaluated predominantly on English, with the commonly used ROUGE metric being English-specific. In this paper, we try to address both issues. We present SumeCzech, a Czech news-based summarization dataset. It contains more than a million documents, each consisting of a headline, a several sentences long abstract and a full text. The dataset can be downloaded using the provided scripts available at http://hdl.handle.net/11234/1-2615. We evaluate several summarization baselines on the dataset, including a strong abstractive approach based on Transformer neural network architecture. The evaluation is performed using a language-agnostic variant of ROUGE. Keywords: SumeCzech, summarization dataset, document summarization, ROUGE, Czech 1. Introduction Similarly to many other NLP tasks, performance of auto- matic document summarization has been improving with the recent rise of neural network methods. While deep neural network models can leverage large datasets, only a few moderately-sized datasets are available for document summarization when compared to, e.g., machine transla- tion. Additionally, document summarization has been explored mostly on English, with the dominant ROUGE metric (Lin, 2004) being English-specific (utilizing English stemmer, stop words and synonyms). In order to provide more data for document summarization in Czech, this paper introduces SumeCzech – a collection of one million Czech news articles, each consisting of a headline, a several sentence abstract and a full text. The documents originate from five Czech Internet news sites. The dataset can be downloaded using our provided scripts. Headline-abstract-text structure of the documents allows the dataset to be used for multiple summarization setups: headline generation either from an abstract or a full text, or generation of a multi-sentence abstract from a full text. To enable automatic evaluation of summarization for Czech, we also propose a straightforward language- agnostic variant of the ROUGE metric, which we call ROUGERAW. We evaluate several baselines for all selected summariza- tion settings. Apart from several unsupervised methods, we evaluate two supervised methods: an extractive one in- spired by approach by Kupiec et al. (1995), and an abstrac- tive baseline based on Transformers neural network archi- tecture (Vaswani et al., 2017). 2. Related Work 2.1. Datasets Sentence summarization has been traditionally connected with the task of headline generation. The task was stan- dardized around the DUC-2003 and DUC-2004 competi- tions (Over et al., 2007), which provided a standard eval- uation set consisting of 500 news articles from New York Times and Associated Press Wire, each paired with 4 dif- ferent human-generated reference summaries. For training, the Gigaword dataset (Graff et al., 2003) has been used frequently, offering 4 million news …

Content truncated to 500 words