Fact-based Content Weighting for Evaluating Abstractive Summarisation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5071–5081 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5071 Fact-based Content Weighting for Evaluating Abstractive Summarisation Xinnuo Xu†, Ondˇrej Duˇsek‡, Jingyi Li†, Verena Rieser† and Ioannis Konstas† †The Interaction Lab, MACS, Heriot-Watt University, Edinburgh, UK ‡Charles University, Faculty of Mathematics and Physics, Prague, Czechia xx6, jl125, v.t.rieser, i.konstas@hw.ac.uk odusek@ufal.mff.cuni.cz Abstract Abstractive summarisation is notoriously hard to evaluate since standard word-overlap-based metrics are biased towards specific words in the human reference. We introduce a new evaluation metric which abstracts away from the word-level and instead is based on fact- level content weighting, i.e. relating the facts of the document to the facts of the summary. We follow the assumption that a good sum- mary will reflect all relevant facts, i.e. the ones present in the ground truth (human-generated reference summary). We confirm this hypoth- esis by showing that our weightings are highly correlated to human perception and compare favourably to the recent manual highlight- based metric of Hardy et al. (2019). 1 Introduction Text summarisation compresses long textual doc- uments into short summaries while retaining the most important information from the source. In contrast to extractive summarisation, which directly copies the most relevant fragments, abstractive summarization retains the most important facts and expresses them via paraphrasing, aggregating and even inferring new facts. Recent advances in neu- ral decoders led to a number of single-document summarisation systems that exhibit some level of abstraction in their outputs, usually in the simplest form of paraphrasing (See et al. (2017); Narayan et al. (2018); Liu and Lapata (2019), inter alia). Evaluating abstractive summarisation remains an open challenge (Schluter, 2017; Kry´sci´nski et al., 2019): First, decoders are amenable to pathoge- niessuch as hallucination and/or omission of im- portant information, which are hard to capture us- ing existing evaluation metrics (Cao et al., 2018; Rohrbach et al., 2018; Duˇsek et al., 2020). Sec- ond, most datasets used for abstractive summari- sation only contain a single reference summary, e.g. (Narayan et al., 2018; V¨olske et al., 2017), which most existing automatic metrics evaluate against, e.g. ROUGE using exact n-gram overlap (Lin, 2004), and thus tend to downvote paraphrases. We propose a new evaluation metric based on content weighting, where we abstract away from the particular surface form of the target summary, but represent it as facts using Semantic Role La- belling (SRL). In this way, we aim to better capture the semantic correctness of a summary, i.e. be more sensitive to hallucinations and omissions.1 In particular, we weight the facts present in the source document according to the facts selected by a human-written summary. This alignment is conducted using contextual, rather than token-level, embeddings, e.g., BERT (Devlin et al., 2019). For evaluation, we measure whether an automatically generated summary is able to capture the same facts as the target. We also show that the com- puted weights correlate well with human percep- tion. Our code is available at https://github. com/XinnuoXu/CorrFA_for_Summarizaion. 2 Related …

Content truncated to 500 words