Is Human Scoring the Best Criteria for Summary Evaluation?

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2184–2191 August 1–6, 2021. ©2021 Association for Computational Linguistics 2184 Is Human Scoring the Best Criteria for Summary Evaluation? Oleg Vasilyev, John Bohannon Primer Technologies Inc. San Francisco, California {oleg,john}@primer.ai Abstract A summary quality measure is judged by how well it correlates with quality scores produced by human annotators. A higher correlation with human scores is considered to be a deci- sive indicator of a better measure. In this work we present observations that cast doubt on this view. We also show a possibility of an alter- native indicator for selecting the best measure from a family of measures, a criterion that does not rely on human scores. 1 Introduction The goal of summarization is to convey important and only important information of the text in a fluent, comprehensible and concise summary, pre- serving the factual consistency with the text. There are several families of automated mea- sures of summary quality. For example, Gabriel et al. (2020) classified the automated measures into four types: question-answering, text recon- struction, semantic similarity and lexical overlap. Each of these types has families of measures, for example SUM-QE (Xenouleas et al., 2019), APES (Eyal et al., 2019), Summa-QA (Scialom et al., 2019), QAEval (Deutsch et al., 2020) and FEQA (Durmus et al., 2020) in question-answering, BLANC (Vasilyev et al., 2020a) in text reconstruc- tion, BERTScore (Zhang et al., 2020), MoverScore (Zhao et al., 2019) and SUPERT (Gao et al., 2020) in semantic similarity, ROUGE (Lin, 2004) and Jensen-Shannon (Louis and Nenkova, 2009) in lex- ical overlap. A high correlation with human evaluation scores is currently accepted as the crucial criterion for choosing a good evaluation measure. Arguably, the factual faithfulness can be annotated objec- tively, with detailed classification of factual errors (Kryscinski et al., 2020; Huang et al., 2020; Vasi- lyev et al., 2020b; Gabriel et al., 2020). However, other summary qualities are subjective; this forces researchers to be careful in design and usage of human annotations (Bhandari et al., 2020; Fabbri et al., 2020; Iskender et al., 2021). Annotation scores depend on the types of texts and on the qual- ification of annotators. For example, there is a big difference in expert and crowd-sourced scores in (Fabbri et al., 2020)1. Annotators are biased in favor of anything that makes the scoring easier: extractiveness of the sum- mary, and focus of the summary on the top part of the document (Ziegler et al., 2020). The annotation process itself differs from how the summary quality is assessed by a typical human reader. A human reader does not have a goal of scoring a summary, but rather uses the summary to guess the content of the text. The contribution of this work: 1. We provide an example of a false ’improve- ment’ of an automated evaluation measure: a dubious modification, imitating a human an- notator behavior, can increase the correlation with human scores. For a contrast, we also provide an example of a true improvement that …

Content truncated to 500 words