NEWTS: A Corpus for News Topic-Focused Summarization

Findings of the Association for Computational Linguistics: ACL 2022, pages 493 - 503 May 22-27, 2022 c⃝2022 Association for Computational Linguistics NEWTS: A Corpus for News Topic-Focused Summarization∗ Seyed Ali Bahrainian1,2, Sheridan Feucht1, and Carsten Eickhoff1 {bahrainian, sheridan_feucht, carsten_eickhoff}@brown.edu 1Department of Computer Science, Brown University, USA 2Machine Learning and Optimization Lab, EPFL, Switzerland Abstract Text summarization models are approaching human levels of fidelity. Existing benchmark- ing corpora provide concordant pairs of full and abridged versions of Web, news or, pro- fessional content. To date, all summariza- tion datasets operate under a one-size-fits-all paradigm that may not reflect the full range of organic summarization needs. Several recently proposed models (e.g., plug and play language models) have the capacity to condition the gen- erated summaries on a desired range of themes. These capacities remain largely unused and un- evaluated as there is no dedicated dataset that would support the task of topic-focused sum- marization. This paper introduces the first topical sum- marization corpus NEWTS, based on the well- known CNN/Dailymail dataset, and annotated via online crowd-sourcing. Each source article is paired with two reference summaries, each focusing on a different theme of the source document. We evaluate a representative range of existing techniques and analyze the effec- tiveness of different prompting methods. 1 Introduction With the recent advances in neural sequence-to- sequence models, the automatic generation of text has reached unparalleled levels of fidelity. Ab- stractive summarization models that aim at generat- ing condensed versions of a source article have outperformed Lead-3 baselines on most bench- mark datasets (See et al., 2017; Lewis et al., 2020). However, all existing summarization benchmarks assume a one-size-fits-all paradigm under which model output is evaluated based on similarity to general-purpose reference summaries reflecting the full content of the original document. While cer- tainly a necessary step, such evaluation approaches might not reflect the full range of summarization ∗ The first author and the second author have an equal contribution. Figure 1: A topical summarization example, summa- rizing a sample document with respect to economy and climate topics. needs anymore. There are manifold settings in which tailored summaries matching the interests of the reader may be required. Some examples in- clude the summarization of complex event streams with a focus on regions, entities or topics of interest for journalists or analysts, understanding reviews or opinions from different perspectives (Hayashi et al., 2021), the summarization of electronic health records with a focus on the medical sub-specialty of the physician reader, or any other form of person- alized summarization targeting explicitly defined or implicitly mined preference parameters. Several recently proposed text generation mod- els already offer the potential of steering the gen- eration process to conform to specific topic dis- tributions (Bahrainian et al., 2021), or sentiment polarity (Shen et al., 2017). Plug and Play Lan- guage Models (PPLM) (Dathathri et al., 2020) let us condition the generation process on themes of interest and text style transfer controls selected at- tributes, such as politeness, emotions, or humor of the …

Content truncated to 500 words