How well do you know your summarization datasets?

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3436–3449 August 1–6, 2021. ©2021 Association for Computational Linguistics 3436 How well do you know your summarization datasets? Priyam Tejaswin ∗ Dhruv Naik Pengfei Liu † Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA {ptejaswi, drn, pliu3}@andrew.cmu.edu Abstract State-of-the-art summarization systems are trained and evaluated on massive datasets scraped from the web. Despite their preva- lence, we know very little about the underly- ing characteristics (data noise, summarization complexity, etc.) of these datasets, and how these affect system performance and the reli- ability of automatic metrics like ROUGE. In this study, we manually analyse 600 samples from three popular summarization datasets. Our study is driven by a six-class typology which captures different noise types (missing facts, entities) and degrees of summarization difficulty (extractive, abstractive). We follow with a thorough analysis of 27 state-of-the-art summarization models and 5 popular metrics, and report our key insights: (1) Datasets have distinct data quality and complexity distribu- tions, which can be traced back to their collec- tion process. (2) The performance of models and reliability of metrics is dependent on sam- ple complexity. (3) Faithful summaries often receive low scores because of the poor diver- sity of references. We release the code, anno- tated data and model outputs.1 1 Introduction The past few years have witnessed major break- throughs and improvements in automatic summa- rization (See et al., 2017; Celikyilmaz et al., 2018; Jadhav and Rajan, 2018; Liu and Lapata, 2019; Liu, 2019; Dou et al., 2020; Yuan et al., 2021; Liu et al., 2021). Apart from the improvements in the sum- marization model architectures (Zhang et al., 2019; Zhong et al., 2020), this growth has been aided by large-scale datasets (Nallapati et al., 2016; Narayan et al., 2018a; Sharma et al., 2019) and automatic evaluation metrics (Lin, 2004; Zhao et al., 2019; ∗This author was the primary contributor. †Corresponding author. 1https://github.com/priyamtejaswin/howwelldoyouknow Kryscinski et al., 2020) which are used for tuning hyperparameters and comparing models. While the reliability of these metrics has been explored exten- sively (Peyrard, 2019; Bhandari et al., 2020; Fabbri et al., 2020), few studies have focused on the un- derlying characteristics of different datasets, and how these impact model performance and metric reliability. Datasets like CNN/DailyMail (Nallapati et al., 2016), Gigaword (Rush et al., 2015), XSum (Narayan et al., 2018a), and many more (Wang and Ling, 2016; Koupaee and Wang, 2018; Kim et al., 2019; Ganesan et al., 2010) were collected by scraping a large collection of web-pages. And for all the benefits this approach offers (seemingly infi- nite samples, diverse subjects, etc) there are some caveats: Data Noise We have no idea about the noise in the dataset. In the context of text summarization, noise could be an incomplete or irrelevant refer- ence. At the moment, its quantity and impact on the performance is unknown. Summarization Complexity What do we really know about the nature of samples in the dataset? Gigaword is a headline generation dataset with short sources …

Content truncated to 500 words