How to Evaluate a Summarizer: Study Design and Statistical Analysis for Manual Linguistic Quality Evaluation

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1861–1875 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1861 How to Evaluate a Summarizer: Study Design and Statistical Analysis for Manual Linguistic Quality Evaluation Julius Steen Katja Markert Department of Computational Linguistics Heidelberg University 69120 Heidelberg, Germany (steen|markert)@cl.uni-heidelberg.de Abstract Manual evaluation is essential to judge progress on automatic text summarization. However, we conduct a survey on recent sum- marization system papers that reveals little agreement on how to perform such evaluation studies. We conduct two evaluation experi- ments on two aspects of summaries’ linguistic quality (coherence and repetitiveness) to com- pare Likert-type and ranking annotations and show that best choice of evaluation method can vary from one aspect to another. In our sur- vey, we also find that study parameters such as the overall number of annotators and distribu- tion of annotators to annotation items are of- ten not fully reported and that subsequent sta- tistical analysis ignores grouping factors aris- ing from one annotator judging multiple sum- maries. Using our evaluation experiments, we show that the total number of annotators can have a strong impact on study power and that current statistical analysis methods can inflate type I error rates up to eight-fold. In addi- tion, we highlight that for the purpose of sys- tem comparison the current practice of elicit- ing multiple judgements per summary leads to less powerful and reliable annotations given a fixed study budget. 1 Introduction Current automatic metrics for summary evaluation have low correlation with human judgements on summary quality, especially for linguistic quality evaluation (Fabbri et al., 2020). As a consequence, manual evaluation is still vital to properly compare the linguistic quality of summarization systems. While the document understanding conferences (DUC) established a standard manual evaluation procedure (Dang, 2005), we conduct a comprehen- sive survey of recent works in text summarization that reveals a wide array of different evaluation questions and methods in current use. Furthermore, DUC procedures were designed for a small set of expert judges, while current evaluation campaigns are often conducted by untrained crowd-workers. The design of the manual annotation, specifically the overall number of annotators as well as the distribution of annotators to annotation items, has substantial impact on power, reliability and type I errors of subsequent statistical analysis. How- ever, most current papers (see Section 2) do not consider the interaction of annotation design and statistical analysis. We investigate the optimal an- notation methods, design and statistical analysis of summary evaluation studies, making the following contributions: 1. We conduct a comprehensive survey on the current practices in manual summary evalu- ation in Section 2. Often, important study parameters, such as the total number of an- notators, are not reported. In addition, sta- tistical significance is either not assessed at all or with tests (t-test or one-way ANOVA) that lead to inflated type I error in the pres- ence of grouping factors (Barr et al., 2013). In summarization evaluation, grouping …

Content truncated to 500 words