CDEvalSumm: An Empirical Study of Cross-Dataset Evaluation for Neural Summarization Systems

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3679–3691 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3679 CDEvalSumm: An Empirical Study of Cross-Dataset Evaluation for Neural Summarization Systems Yiran Chen∗, Pengfei Liu♯∗, Ming Zhong, Zi-Yi Dou♯, Danqing Wang, Xipeng Qiu†, Xuanjing Huang Shanghai Key Laboratory of Intelligent Information Processing, Fudan University School of Computer Science, Fudan University 2005 Songhu Road, Shanghai, China ♯Carnegie Mellon University {yrchen19,mzhong18,dqwang18,xpqiu,xjhuang}@fudan.edu.cn {zdou,pliu3}@cs.cmu.edu Abstract Neural network-based models augmented with unsupervised pre-trained knowledge have achieved impressive performance on text sum- marization. However, most existing evaluation methods are limited to an in-domain setting, where summarizers are trained and evaluated on the same dataset. We argue that this ap- proach can narrow our understanding of the generalization ability for different summariza- tion systems. In this paper, we perform an in- depth analysis of characteristics of different datasets and investigate the performance of dif- ferent summarization models under a cross- dataset setting, in which a summarizer trained on one corpus will be evaluated on a range of out-of-domain corpora. A comprehensive study of 11 representative summarization sys- tems on 5 datasets from different domains re- veals the effect of model architectures and generation ways (i.e. abstractive and extrac- tive) on model generalization ability. Further, experimental results shed light on the limita- tions of existing summarizers. Brief introduc- tion and supplementary code can be found in https://github.com/zide05/CDEvalSumm. 1 Introduction Neural summarizers have achieved impressive per- formance when evaluated by ROUGE (Lin, 2004) on in-domain setting, and the recent success of pre- trained models drives the state-of-the-art results on benchmarks to a new level (Liu and Lapata, 2019; Liu, 2019; Zhong et al., 2019a; Zhang et al., 2019; Lewis et al., 2019; Zhong et al., 2020). How- ever, the superior performance is not a guarantee of a perfect system since exsiting models tend to show defects when evaluated from other aspects. For example, Zhang et al. (2018) observes that ∗These two authors contributed equally. †Corresponding author. Figure 1: Ranking (descending order) of current 11 top-scoring summarization systems (Abstractive mod- els are red while extractive ones are blue). Each sys- tem is evaluated based on three diverse evaluation meth- ods: (a) averaging each system’s in-dataset ROUGE-2 F1 scores (R2) over five datasets; (b-c) evaluating sys- tems using our designed cross-dataset measures: stiff- R2, stable-R2 (Sec. 5). Notably, BERTmatch and BART are two state-of-the-art models for extractive and ab- stractive summarization respectively (highlighted by blue and red boxes). many abstractive systems tend to be near-extractive in practice. Cao et al. (2018); Wang et al. (2020); Kry´sci´nski et al. (2019); Maynez et al. (2020) re- veal that most generated summaries are factually in- correct. These non-mainstream evaluation methods make it easier to identify the model’s weaknesses. Orthogonal to above two evaluation aspects, we aim to diagnose the limitation of existing systems under cross-dataset evaluation, in which a sum- marization system trained on one corpus would be evaluated on a range of out-of-dataset corpora. Instead of evaluating the quality of summarizers …

Content truncated to 500 words