BUMP: A Benchmark of Unfaithful Minimal Pairs for Meta-Evaluation of Faithfulness Metrics

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12788–12812 July 9-14, 2023 ©2023 Association for Computational Linguistics BUMP: A Benchmark of Unfaithful Minimal Pairs for Meta-Evaluation of Faithfulness Metrics Liang Ma1 Shuyang Cao2 Robert L. Logan IV1 Di Lu1 Shihao Ran1 Ke Zhang1 Joel Tetreault1 Alejandro Jaimes1 1Dataminr Inc. 2University of Michigan, Ann Arbor {lma,rlogan,dlu,sran,kzhang,jtetreault, ajaimes}@dataminr.com caoshuy@umich.edu Abstract The proliferation of automatic faithfulness met- rics for summarization has produced a need for benchmarks to evaluate them. While exist- ing benchmarks measure the correlation with human judgements of faithfulness on model- generated summaries, they are insufficient for diagnosing whether metrics are: 1) consistent, i.e., indicate lower faithfulness as errors are introduced into a summary, 2) effective on human-written texts, and 3) sensitive to dif- ferent error types (as summaries can contain multiple errors). To address these needs, we present a benchmark of unfaithful minimal pairs (BUMP), a dataset of 889 human-written, minimally different summary pairs, where a sin- gle error is introduced to a summary from the CNN/DailyMail dataset to produce an unfaith- ful summary. We find BUMP complements existing benchmarks in a number of ways: 1) the summaries in BUMP are harder to discrimi- nate and less probable under SOTA summariza- tion models, 2) unlike non-pair-based datasets, BUMP can be used to measure the consistency of metrics, and reveals that the most discrimi- native metrics tend not to be the most consis- tent, and 3) unlike datasets containing gener- ated summaries with multiple errors, BUMP enables the measurement of metrics’ perfor- mance on individual error types. 1 Introduction Although modern abstractive summarization sys- tems have improved in their ability to produce flu- ent text (Lewis et al., 2020), their ability to gen- erate text that is factually grounded in the source article remains an issue (Kryscinski et al., 2020). This phenomenon has inspired the NLP community to develop faithfulness evaluation metrics (Fabbri et al., 2022; Laban et al., 2022; Honovich et al., 2021; Scialom et al., 2021) that automatically mea- sure the extent to which abstractive summarization systems produce summaries that contain informa- tion that cannot be verified by the source article. As the number of these automatic faithfulness metrics has increased, there has arisen a corre- sponding need for benchmarks that evaluate their relative strengths. To satisfy this need, researchers have developed datasets such as FRANK (Pagnoni et al., 2021) and TRUE (Honovich et al., 2022) that are comprised of model-generated summaries along with human-annotated faithfulness levels. Al- though these datasets are useful for evaluating the degree to which faithfulness metrics correlate with human judgements and can discriminate unfaith- ful summaries, a number of factors limit the con- clusions that can be drawn from them. For one, because model summaries can vary in terms of length, content, and number of errors, these bench- marks are ill-suited for drawing conclusions about the consistency (Gabriel et al., 2021) of metrics, i.e., whether their scores indicate lower faithful- ness as summaries become increasingly …

Content truncated to 500 words