Intrinsic Evaluation of Summarization Datasets

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8075–8096, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8075 Intrinsic Evaluation of Summarization Datasets Rishi Bommasani Department of Computer Science Stanford University nlprishi@cs.stanford.edu Claire Cardie Department of Computer Science Cornell University cardie@cs.cornell.edu Abstract High quality data forms the bedrock for build- ing meaningful statistical models in NLP. Con- sequently, data quality must be evaluated ei- ther during dataset construction or post hoc. Almost all popular summarization datasets are drawn from natural sources and do not come with inherent quality assurance guarantees. In spite of this, data quality has gone largely unquestioned for many recent summarization datasets. We perform the first large-scale eval- uation of summarization datasets by introduc- ing 5 intrinsic metrics and applying them to 10 popular datasets. We find that data usage in recent summarization research is sometimes inconsistent with the underlying properties of the datasets employed. Further, we discover that our metrics can serve the additional pur- pose of being inexpensive heuristics for detect- ing generically low quality examples. 1 Introduction Data understanding is fundamentally important in natural language processing (NLP); for data-driven learning-based methods (e.g. neural networks), the quality of the training data bounds the quality of models learned using it. Therefore, understanding this data is necessary in order to ensure that models learn to perform a given task correctly. Understanding data is a multidimensional prob- lem. One line of inquiry has demonstrated why prominent datasets are insufficiently challenging: many data examples can be solved by alternative heuristics that do not encode an approach that is faithful to the task (McCoy et al., 2019). From the perspective of datasets, several works have shown that standard datasets in areas such as visual question answering (Zhang et al., 2016; Kafle and Kanan, 2017), natural language inference (Guru- rangan et al., 2018; Poliak et al., 2018), and reading comprehension (Kaushik and Lipton, 2018) con- tain annotation artifacts that often give rise to these spurious correlations or reasoning shortcuts. Data understanding can also inform scientific and ethi- cal decision-making (Bender and Friedman, 2018; Gebru et al., 2018; Mitchell et al., 2019) with re- cent work studying how social biases encoded in training data propagate to learned models (Zhao et al., 2019; Tan and Celis, 2019). In this work, we extend these efforts towards the setting of summarization. We find this to be partic- ularly timely since several summarization datasets have been released in recent years with little dis- cussion of data quality. While prior work on evalu- ating NLP datasets has focused on their difficulty, transparency, or bias, we consider broadly the over- all quality of the dataset — in our case, for the task of summarization.1 Our central insight is that desirable properties of a summary can be readily estimated by adapting and applying existing NLP methods. With this in mind, we present a multi- aspect large-scale study of summarization datasets that dissects summarization into 5 properties that are evaluated across 10 datasets spanning multiple …

Content truncated to 500 words