Studying Summarization Evaluation Metrics in the Appropriate Scoring Range

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5093–5100 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5093 Studying Summarization Evaluation Metrics in the Appropriate Scoring Range Maxime Peyrard∗ EPFL maxime.peyrard@epfl.ch Abstract In summarization, automatic evaluation met- rics are usually compared based on their abil- ity to correlate with human judgments. Un- fortunately, the few existing human judgment datasets have been created as by-products of the manual evaluations performed during the DUC/TAC shared tasks. However, modern systems are typically better than the best sys- tems submitted at the time of these shared tasks. We show that, surprisingly, evalua- tion metrics which behave similarly on these datasets (average-scoring range) strongly dis- agree in the higher-scoring range in which cur- rent systems now operate. It is problematic because metrics disagree yet we can’t decide which one to trust. This is a call for collect- ing human judgments for high-scoring sum- maries as this would resolve the debate over which metrics to trust. This would also be greatly beneficial to further improve summa- rization systems and metrics alike. 1 Introduction The progress in summarization is tightly inter- twined with the capability to quickly measure im- provements. Thus, a significant body of research was dedicated to the development of automatic metrics (Lloret et al., 2018). Yet, this remains an open problem (Rankel et al., 2013). Typically, evaluation metrics are compared based on their ability to correlate with humans (Lin and Hovy, 2003). Then, the selected met- rics heavily influence summarization research by guiding progress (Lloret et al., 2018) and by pro- viding supervision for training summarization sys- tems (Yogan et al., 2016). Despite their central role, few human judgment datasets have been created. The existing ones are the result of the manual evaluations performed ∗Research partly done at UKP Lab from TU Darmstadt Figure 1: The blue distribution represents the score dis- tribution of summaries available in the human judg- ment datasets of TAC-2008 and TAC-2009. The red distribution is the score distribution of summaries gen- erated by mordern systems. The green distribution cor- responds to the score distribution of summaries we gen- erated in this work as described in section 3. during shared tasks (Dang and Owczarzak, 2008, 2009). Thus, the annotated summaries are mostly aver- age compared to nowadays standards. Indeed, the best systems submitted at the time of these shared- tasks have typically served as baselines for subse- quent works. This is illustrated by figure 1, which compares the score distribution of summaries in human judgment datasets with the score distri- bution of modern summarization systems.1 The score distribution on which evaluation metrics are tested (blue zone) differs from the one in which they now operate (red zone). Thus, there is no guarantee that evaluation metrics behave accord- ing to human judgments in the high-scoring range. Yet, summarization systems explicitly target high- scoring summaries (Radev et al., 2003). In this work, we study several evaluation met- rics in this high-scoring range …

Content truncated to 500 words