Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies

Proceedings of NAACL-HLT 2018, pages 708–719 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics NEWSROOM: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies Max Grusky1,2, Mor Naaman2, Yoav Artzi1,2 1Department of Computer Science, 2Cornell Tech Cornell University, New York, NY 10044 {grusky@cs, mor@jacobs, yoav@cs}.cornell.edu Abstract We present NEWSROOM, a summarization dataset of 1.3 million articles and summaries written by authors and editors in newsrooms of 38 major news publications. Extracted from search and social media metadata between 1998 and 2017, these high-quality summaries demonstrate high diversity of summarization styles. In particular, the summaries combine abstractive and extractive strategies, borrow- ing words and phrases from articles at vary- ing rates. We analyze the extraction strategies used in NEWSROOM summaries against other datasets to quantify the diversity and difficulty of our new data, and train existing methods on the data to evaluate its utility and challenges. The dataset is available online at summari.es. 1 Introduction The development of learning methods for auto- matic summarization is constrained by the lim- ited high-quality data available for training and evaluation. Large datasets have driven rapid im- provement in other natural language generation tasks, such as machine translation, where data size and diversity have proven critical for mod- eling the alignment between source and target texts (Tiedemann, 2012). Similar challenges exist in summarization, with the additional complica- tions introduced by the length of source texts and the diversity of summarization strategies used by writers. Access to large-scale high-quality data is an essential prerequisite for making substan- tial progress in summarization. In this paper, we present NEWSROOM, a dataset with 1.3 million news articles and human-written summaries. NEWSROOM’s summaries were written by au- thors and editors in the newsrooms of news, sports, entertainment, financial, and other publications. The summaries were published with articles as HTML metadata for social media services and Abstractive Summary: South African photographer Anton Hammerl, missing in Libya since April 4th, was killed in Libya more than a month ago. Mixed Summary: A major climate protest in New York on Sunday could mark a seminal shift in the politics of global warming, just ahead of the U.N. Climate Summit. Extractive Summary: A person familiar with the search tells The Associated Press that Texas has offered its head coaching job to Louisvilles Charlie Strong and he is expected to accept. Figure 1: NEWSROOM summaries showing different extraction strategies, from time.com, mashable.com, and foxsports.com. Multi-word phrases shared be- tween article and summary are underlined. Novel words used only in the summary are italicized. search engines page descriptions. NEWSROOM summaries are written by humans, for common readers, and with the explicit purpose of summa- rization. As a result, NEWSROOM is a nearly two decade-long snapshot representing how single- document summarization is used in practice across a variety of sources, writers, and topics. Identifying large, high-quality resources for summarization has called for creative solutions in the past. This includes using news head- lines as summaries of article prefixes (Napoles et al., …

Content truncated to 500 words