Neural Text Summarization: A Critical Evaluation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 540–551, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 540 Neural Text Summarization: A Critical Evaluation Wojciech Kry´sci´nski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, Richard Socher Salesforce Research {kryscinski,nkeskar,bmccann,cxiong,rsocher}@salesforce.com Abstract Text summarization aims at compressing long documents into a shorter form that conveys the most important parts of the original docu- ment. Despite increased interest in the com- munity and notable research effort, progress on benchmark datasets has stagnated. We crit- ically evaluate key ingredients of the current research setup: datasets, evaluation metrics, and models, and highlight three primary short- comings: 1) automatically collected datasets leave the task underconstrained and may con- tain noise detrimental to training and evalua- tion, 2) current evaluation protocol is weakly correlated with human judgment and does not account for important characteristics such as factual correctness, 3) models overfit to layout biases of current datasets and offer limited di- versity in their outputs. 1 Introduction Text summarization aims at compressing long tex- tual documents into a short, human readable form that contains the most important information from the source. Two strategies of generating sum- maries are extractive (Dorr et al., 2003; Nallapati et al., 2017), where salient fragments of the source document are identified and directly copied into the summary, and abstractive (Rush et al., 2015; See et al., 2017), where the salient parts are de- tected and paraphrased to form the final output. The number of summarization models intro- duced every year has been increasing rapidly. Advancements in neural network architec- tures (Sutskever et al., 2014; Bahdanau et al., 2015; Vinyals et al., 2015; Vaswani et al., 2017) and the availability of large scale data (Sandhaus, 2008; Nallapati et al., 2016a; Grusky et al., 2018) enabled the transition from systems based on expert knowledge and heuristics to data-driven approaches powered by end-to-end deep neural models. Current approaches to text summa- rization utilize advanced attention and copying mechanisms (See et al., 2017; Tan et al., 2017; Cohan et al., 2018), multi-task and multi-reward training techniques (Guo et al., 2018; Pasunuru and Bansal, 2018; Kry´sci´nski et al., 2018), rein- forcement learning strategies (Paulus et al., 2017; Narayan et al., 2018b; Dong et al., 2018; Wu and Hu, 2018), and hybrid extractive-abstractive models (Liu et al., 2018; Hsu et al., 2018; Gehrmann et al., 2018; Chen and Bansal, 2018). Many of the introduced models are trained on the CNN/DailyMail (Nallapati et al., 2016a) news corpus, a popular benchmark for the field, and are evaluated based on n-gram overlap between the generated and target summaries with the ROUGE package (Lin, 2004). Despite substantial research effort, the progress on these benchmarks has stagnated. State-of-the- art models only slightly outperform the Lead-3 baseline, which generates summaries by extract- ing the first three sentences of the source doc- ument. We argue that this stagnation can be partially attributed to the current research setup, which …

Content truncated to 500 words