A Data Set for the Analysis of Text Quality Dimensions in Summarization Evaluation

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6690–6699 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6690 A Data Set for the Analysis of Text Quality Dimensions in Summarization Evaluation Margot Mieskes1, Eneldo Loza Menc´ıa2, Tim Kronsbein2 1 Darmstadt University of Applied Sciences, Germany 2 Knowledge Engineering Group, Technische Universit¨at Darmstadt, Germany margot.mieskes@h-da.de, research@eneldo.net, tim-kronsbein@gmx.de Abstract Automatic evaluation of summarization focuses on developing a metric to represent the quality of the resulting text. However, text quality is represented in a variety of dimensions ranging from grammaticality to readability and coherence. In our work, we analyze the depen- dencies between a variety of quality dimensions on automatically created multi-document summaries and which dimensions automatic evaluation metrics such as ROUGE, PEAK or JSD are able to capture. Our results indicate that variants of ROUGE are correlated to various quality dimensions and that some automatic summarization methods achieve higher quality summaries than others with respect to individual summary quality dimensions. Our results also indicate that differentiating between quality dimensions facilitates inspection and fine-grained comparison of summarization methods and its characteristics. Keywords: Text Quality, Summarization Evaluation, Multi-document Summarization Data Set 1. Introduction Work on text summarization evaluation focused strongly on creating automatic methods for evaluation, which corre- late well with human judgements. Early work on ROUGE showed a good correlation between ROUGE 1 and manual quality evaluations (Lin and Hovy, 2003). But those evalua- tions focused on only few quality dimensions. Text quality has to be measured in a range of dimensions, which cap- ture various aspects of a text. The question that arises is, which of the various quality dimensions are represented by the various automatic evaluation metrics when used without reference summaries. An additional question is what type of automatic evaluation method is best suited for summa- rization of large-scale heterogeneous data sets and which method achieves the highest scores on the various quality dimensions. We aim at answering these questions by a set of experiments in order to determine in what way various quality dimensions are represented by evaluation measures without reference summaries and which automatic summa- rization method gives good results on the various quality dimensions on a large-scale heterogeneous data set. Our contributions are as follows: • A data set of manual annotations of summaries – us- ing both Likert scale evaluations as well as pairwise comparisons – of a range of various automatic sum- marization methods. • An analysis on both evaluations with respect to the various quality dimensions and the various summa- rization systems. • We make the set of evaluations and automatic sum- maries available to the research community.1 This can, e.g., used to train evaluation systems. To the best of our knowledge, it is the first data set of judgements of automatic multi-document summarization 1Available at https://github.com/keelm/DIP-SumEval and https://tudatalib.ulb.tu-darmstadt.de/handle/tudatalib/2303. systems on a large variety of quality dimensions. 2. Related Work This work draws on a range of areas and previous work in automatic summarization. …

Content truncated to 500 words