GeSERA: General-domain Summary Evaluation by Relevance Analysis

Proceedings of Recent Advances in Natural Language Processing, pages 856–867 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_098 856 GeSERA: General-domain Summary Evaluation by Relevance Analysis Jessica L´opez Espejel∗†, Ga¨el de Chalendar∗, Jorge Garcia Flores†, Thierry Charnois†, and Ivan Vladimir Meza Ruiz‡ ∗ Universit´e Paris-Saclay, CEA, List, F-91120, Palaiseau, France gael.de-chalendar@cea.fr, † CNRS-LIPN-Universit´e Sorbonne Paris Nord, France {jessicanayeli.lopezespejel, charnois}@lipn.univ-paris13.fr ; jgflores@lipn.fr ‡ IIMAS, UNAM, Mexico ivanvladimir@turing.iimas.unam.mx Abstract We present GeSERA, an open-source im- proved version of SERA for evaluating au- tomatic extractive and abstractive summaries from the general domain. SERA is based on a search engine that compares candidate and reference summaries (called queries) against an information retrieval document base (called index). SERA was originally designed for the biomedical domain only, where it showed a better correlation with manual methods than the widely used lexical-based ROUGE method. In this paper, we take out SERA from the biomedical domain to the general one by adapting its content-based method to suc- cessfully evaluate summaries from the gen- eral domain. First, we improve the query reformulation strategy with POS Tags analy- sis of general-domain corpora. Second, we replace the biomedical index used in SERA with two article collections from AQUAINT-2 and Wikipedia. We conduct experiments with TAC2008, TAC2009, and CNNDM datasets. Results show that, in most cases, GeSERA achieves higher correlations with manual eval- uation methods than SERA, while it reduces its gap with ROUGE for general-domain sum- mary evaluation. GeSERA even surpasses ROUGE in two cases of TAC2009. Finally, we conduct extensive experiments and provide a comprehensive study of the impact of hu- man annotators and the index size on summary evaluation with SERA and GeSERA. 1 Introduction Automatic summary evaluation is a challenging task in Natural Language Processing (NLP). Eval- uation is usually done by humans, but manual evaluation is subjective, costly and time expen- sive (Lin and Hovy, 2002). Automatic evaluation methods (Lin, 2004a; Torres-Moreno et al., 2010; Zhao et al., 2019; Zhang et al., 2020) are an alter- native to save time for users who extract the most relevant content from the web using Automatic Text Summarization systems (ATS). There exist two types of evaluation approaches: (1) manual evaluation methods like Pyramid (Nenkova and Passonneau, 2004) and Responsiveness, where hu- man intervention is mandatory, and (2) automatic evaluation methods, where human intervention can be needed as a ground-truth reference (Lin, 2004a; Cohan and Goharian, 2016) or not (Torres- Moreno et al., 2010; Cabrera-Diego and Torres- Moreno, 2018). Summary Evaluation by Relevance Analysis (SERA) (Cohan and Goharian, 2016) is an auto- matic evaluation method that partially relies on hu- man references to evaluate abstractive summaries from the biomedical domain. It was proposed as an alternative to ROUGE (Lin, 2004a), the widely used automatic metric, that is based on lexical overlaps between candidate and reference sum- maries. ROUGE is unfair to evaluate abstractive summaries where the ATS paraphrases the text in- stead of just copying-pasting chunks of it (Cohan and Goharian, 2016). SERA is based on content relevance and is fairer to evaluate abstractive summaries …

Content truncated to 500 words