Measuring Faithfulness of Abstractive Summaries

Measuring Faithfulness of Abstractive Summaries Tim Fischer Steffen Remus Chris Biemann Language Technology Group Department of Informatics Universität Hamburg, Germany {firstname.lastname}@uni-hamburg.de Abstract Recent abstractive summarization systems fail to generate factually consistent – faithful – sum- maries, which heavily limits their practical ap- plication. Commonly, these models tend to mix concepts from the source or hallucinate new content, completely ignoring the source. Ad- dressing the faithfulness problem is perhaps the most critical challenge for current abstractive summarization systems. First automatic faith- fulness metrics were proposed, but we argue that existing methods do not yet utilize the full potential that this field has to offer and intro- duce new approaches to assess factual correct- ness. We evaluate existing and our proposed methods by correlating them with human judge- ments and find that BERTScore works well. Fi- nally, we conduct a qualitative and quantitative error analysis, which reveals common prob- lems and indicates means to further improve the metrics. 1 Introduction Abstractive summarization is the task of generating an informative and fluent summary that is faith- ful to the source document. Recent progress in neural text generation has led to significant im- provements and well-performing state-of-the-art abstractive summarization systems (Zhang et al., 2019; Lewis et al., 2020). Despite these advances, recent models fail to meet one of the essential re- quirements of practical summarization systems: in- formation of a generated summary must match the facts of the source document. We follow Cao et al. (2018) and refer to this aspect as faithfulness in this work. Recent studies have shown that around 30% of automatically generated summaries from neural summarization systems contain unfaithful information (Cao et al., 2018; Falke et al., 2019; Kryscinski et al., 2019), especially when a sentence combines content from multiple source sentences (Lebanoff et al., 2019). Table 1 shows a misleading and unfaithful summary demonstrating this issue. Source The restaurant began serving puppy platters after a new law was introduced allowing dogs to eat at restaurants – as long as they were outdoors! Summary New rules have come into place that you can eat your dog. Table 1: A generated, unfaithful summary found in the XSUM hallucination dataset by Maynez et al. (2020). Researchers identified multiple challenges for developing faithful systems. One challenge is eval- uation, as current automatic metrics are inadequate. Typical metrics like ROUGE (Lin, 2004), BLEU (Papineni et al., 2002) or METEOR (Banerjee and Lavie, 2005) are insensitive to semantic errors. These n-gram-based approaches weight all portions of the text equally, even when only a small fraction of the n-grams carry most of the semantic content. Consequently, factual inconsistencies caused by small changes are overshadowed by high n-gram overlaps. Another challenge is the optimization of abstractive models. Generating summaries that highly overlap with human references does not guarantee faithful summaries (Zhang et al., 2020b). Initial work on metrics to automatically assess faithfulness will be discussed in Section 2 and 3, however, no consensus has been reached to date. We argue that the currently available means to au- …

Content truncated to 500 words