LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1650–1669 May 2-6, 2023 ©2023 Association for Computational Linguistics LONGEVAL: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization Kalpesh Krishna♠ ∗ Erin Bransom♢ Bailey Kuehl♢ Mohit Iyyer♠ Pradeep Dasigi♢ Arman Cohan♢♡ Kyle Lo♢ ♠University of Massachusetts Amherst, ♢Allen Institute for AI, ♡Yale University {kalpesh,miyyer}@cs.umass.edu {erinbransom,baileyk,pradeepd,armanc,kylel}@allenai.org Abstract While human evaluation remains best prac- tice for accurately judging the faithfulness of automatically-generated summaries, few solu- tions exist to address the increased difficulty and workload when evaluating long-form sum- maries. Through a survey of 162 papers on long-form summarization, we first shed light on current human evaluation practices surround- ing long-form summaries. We find that 73% of these papers do not perform any human eval- uation on model-generated summaries, while other works face new difficulties that manifest when dealing with long documents (e.g., low inter-annotator agreement). Motivated by our survey, we present LONGEVAL, a set of guide- lines for human evaluation of faithfulness in long-form summaries that addresses the fol- lowing challenges: (1) How can we achieve high inter-annotator agreement on faithfulness scores? (2) How can we minimize annotator workload while maintaining accurate faithful- ness scores? and (3) Do humans benefit from automated alignment between summary and source snippets? We deploy LONGEVAL in an- notation studies on two long-form summariza- tion datasets in different domains (SQuALITY and PubMed), and we find that switching to a finer granularity of judgment (e.g., clause-level) reduces inter-annotator variance in faithfulness scores (e.g., std-dev from 18.5 to 6.8). We also show that scores from a partial annotation of fine-grained units highly correlates with scores from a full annotation workload (0.89 Kendall’s τ using 50% judgments). We release our hu- man judgments, annotation templates, and our software for future research.1 1 Introduction Human judgments are considered the gold standard for evaluating model-generated sum- 1https://github.com/martiansideofthemoon/ longeval-summarization *Work done during in an internship at AI2. Details of individual author contributions can be found here. maries (Kryscinski et al., 2019; Fabbri et al., 2021) and generated text more broadly (Celikyilmaz et al., 2020). Unfortunately, human evaluation tends to be labor-intensive, expensive to scale, and diffi- cult to design. This is problematic as a large num- ber of judged examples is needed to draw statisti- cally significant conclusions about system perfor- mances (Wei and Jia, 2021) or correlations between human judgments and automatic metrics (Deutsch et al., 2021). Human evaluation is especially chal- lenging when long sequences of generated text need to be evaluated, due to the inherent subjectiv- ity in the task (Karpinska et al., 2021; Clark et al., 2021; Krishna et al., 2021; Goyal et al., 2022). To better understand the challenges of human evaluation on long-form summaries (150 words or longer), we first conduct a comprehensive sur- vey of 162 publications and preprints on long-form summarization (Section 2). We find that 119 pa- pers (73%) do not perform human evaluation on long-form summaries, while the remaining papers deviate significantly from suggested best practices …

Content truncated to 500 words