Enumeration of Extractive Oracle Summaries

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 386–396, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics Enumeration of Extractive Oracle Summaries Tsutomu Hirao and Masaaki Nishino and Jun Suzuki and Masaaki Nagata NTT Communication Science Laboratories, NTT Corporation 2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0237, Japan {hirao.tsutomu,nishino.masaaki}@lab.ntt.co.jp {suzuki.jun,nagata.masaaki}@lab.ntt.co.jp Abstract To analyze the limitations and the future directions of the extractive summarization paradigm, this paper proposes an Inte- ger Linear Programming (ILP) formula- tion to obtain extractive oracle summaries in terms of ROUGEn. We also propose an algorithm that enumerates all of the ora- cle summaries for a set of reference sum- maries to exploit F-measures that evalu- ate which system summaries contain how many sentences that are extracted as an or- acle summary. Our experimental results obtained from Document Understanding Conference (DUC) corpora demonstrated the following: (1) room still exists to improve the performance of extractive summarization; (2) the F-measures de- rived from the enumerated oracle sum- maries have significantly stronger correla- tions with human judgment than those de- rived from single oracle summaries. 1 Introduction Recently, compressive and abstractive summariza- tion are attracting attention (e.g., Almeida and Martins (2013), Qian and Liu (2013), Yao et al. (2015), Banerjee et al. (2015), Bing et al. (2015)). However, extractive summarization remains a pri- mary research topic because the linguistic qual- ity of the resultant summaries is guaranteed, at least at the sentence level, which is a key require- ment for practical use (e.g., Hong and Nenkova (2014), Hong et al. (2015), Yogatama et al. (2015), Parveen et al. (2015)). The summarization research community is ex- periencing a paradigm shift from extractive to compressive or abstractive summarization. Cur- rently our question is: “Is extractive summariza- tion still useful research?” To answer it, the ul- timate limitations of the extractive summarization paradigm must be comprehended; that is, we have to determine its upper bound and compare it with the performance of the state-of-the-art summariza- tion methods. Since ROUGEn is the de-facto auto- matic evaluation method and is employed in many text summarization studies, an oracle summary is defined as a set of sentences that have a maximum ROUGEn score. If the ROUGEn score of an or- acle summary outperforms that of a system that employs another summarization approach, the ex- tractive summarization paradigm is worthwhile to leverage research resources. As another benefit, identifying an oracle sum- mary for a set of reference summaries allows us to utilize yet another evaluation measure. Since both oracle and extractive summaries are sets of sen- tences, it is easy to check whether a system sum- mary contains sentences in the oracle summary. As a result, F-measures, which are available to evaluate a system summary, are useful for evaluat- ing classification-based extractive summarization (Mani and Bloedorn, 1998; Osborne, 2002; Hi- rao et al., 2002). Since ROUGEn evaluation does not identify which sentence is important, an F- measure conveys useful information in terms of “important …

Content truncated to 500 words