Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluations
38193Authors
nan
Abstract
AbstractEvaluating multi-document summarization (MDS) quality is difficult. This is especially true in the case of MDS for biomedical literature reviews, where models must synthesize contradicting evidence reported across different documents. Prior work has shown that rather than performing the task, models may exploit shortcuts that are difficult to detect using standard n-gram similarity metrics such as ROUGE. Better automated evaluation metrics are needed, but few resources exist to assess metrics when they are proposed. Therefore, we introduce a dataset of human-assessed summary quality facets and pairwise preferences to encourage and support the development of better automated evaluation methods for literature review MDS. We take advantage of community submissions to the Multi-document Summarization for Literature Review (MSLR) shared task to compile a diverse and representative sample of generated summaries. We analyze how automated summarization evaluation metrics correlate with lexical features of generated summaries, to other automated metrics including several we propose in this work, and to aspects of human-assessed summary quality. We find that not only do automated metrics fail to capture aspects of quality as assessed by humans, in many cases the system rankings produced by these metrics are anti-correlated with rankings according to human annotators.
Keywords
Full Content
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 950–966 July 9-14, 2023 ©2023 Association for Computational Linguistics Subjective Crowd Disagreements for Subjective Data: Uncovering Meaningful CrowdOpinion with Population-level Learning Tharindu Cyril Weerasooriya 1*, Sarah Luger2, Saloni Poddar1, Ashiqur R. KhudaBukhsh1, Christopher M. Homan1 1Rochester Institute of Technology, USA 2Orange Silicon Valley *cyriltcw@gmail.com Abstract This paper contains content that can be offen- sive or disturbing. Human-annotated data plays a critical role in the fairness of AI systems, including those that deal with life-altering decisions or moderat- ing human-created web/social media content. Conventionally, annotator disagreements are resolved before any learning takes place. How- ever, researchers are increasingly identifying annotator disagreement as pervasive and mean- ingful. They also question the performance of a system when annotators disagree. Partic- ularly when minority views are disregarded, especially among groups that may already be underrepresented in the annotator population. In this paper, we introduce CrowdOpinion, an unsupervised learning based approach that uses language features and label distributions to pool similar items into larger samples of label dis- tributions. We experiment with four generative and one density-based clustering method, ap- plied to five linear combinations of label dis- tributions and features. We use five publicly available benchmark datasets (with varying lev- els of annotator disagreements) from social me- dia (Twitter, Gab, and Reddit). We also experi- ment in the wild using a dataset from Facebook, where annotations come from the platform it- self by users reacting to posts. We evaluate CrowdOpinion as a label distribution prediction task using KL-divergence and a single-label problem using accuracy measures. 1 Introduction Long term exposure to offensive, threatening, and hate speech posts through any public-facing social media platform can lead to depression or even phys- ical injuries, specially at a younger age (Pedalino and Camerini, 2022). This is a persistent problem in social and web content where the impact could be not limited to just the targeted parties but expand to anyone in the community consuming the content Intended Not Intd. Prob. Intd. Not Prob. Intd. 0.0 0.5 1.0 1.5 2.0 Count (a) DSIE1 “Dur- ing the Thanksgiv- ing season, many Americans support Jennie-O-cide.” Intended Not Intd. Prob. Intd. Not Prob. Intd. 0.0 0.5 1.0 1.5 2.0 Count (b) DSIE2 “That’s not your real name. You’re supposed to have a foreign name or something.” Intended Not Intd. Prob. Intd. Not Prob. Intd. 0 1 2 3 Count (c) DSIE3 “what’s the difference between jelly and jam? i can’t jelly my d*** down your throat.” Figure 1: Examples from DSI (Sap et al., 2019), from human annotation for Twitter posts on whether they are intended to be offensive. These examples show how offense cannot generalize, and in cases when a majority of the annotators are not offended the input for a classifier is the majority voice. (Benson, 1996; Fauman, 2008; Chandrasekharan et al., 2017; Müller and Schwarz, 2020). Language used by content creators in social me- dia (see Figure 1) with a subtle …
Content truncated to 500 words
Labeled Text
AbstractEvaluating multi-document summarization (MDS) quality is difficult. This is especially true in the case of MDS for biomedical literature reviews, where models must synthesize contradicting evidence reported across different documents. Prior work has shown that rather than performing the task, models may exploit shortcuts that are difficult to detect using standard n-gram similarity metrics such as ROUGE. Better automated evaluation metrics are needed, but few resources exist to assess metrics when they are proposed. Therefore, we introduce a dataset of human-assessed summary quality facets and pairwise preferences to encourage and support the development of better automated evaluation methods for literature review MDS. We take advantage of community submissions to the Multi-document Summarization for Literature Review (MSLR) shared task to compile a diverse and representative sample of generated summaries. We analyze how automated summarization evaluation metrics correlate with lexical features of generated summaries, to other automated metrics including several we propose in this work, and to aspects of human-assessed summary quality. We find that not only do automated metrics fail to capture aspects of quality as assessed by humans, in many cases the system rankings produced by these metrics are anti-correlated with rankings according to human annotators.
Publication Details
- Paper ID:
- 38193
- Year:
- 2023
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan