Language Agnostic Automatic Summarization Evaluation

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6656–6662 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6656 Language Agnostic Automatic Summarization Evaluation Christopher Tauchmann⋆, Margot Mieskes‡ ⋆Artificial Intelligence and Machine Learning Lab, ‡Information Science, ⋆Technische Universit¨at Darmstadt, ‡Darmstadt University of Applied Sciences ⋆tauchmann@cs.tu-darmstadt.de, ‡margot.mieskes@h-da.de Abstract So far work on automatic summarization has dealt primarily with English data. Accordingly, evaluation methods were primarily de- veloped with this language in mind. In our work, we present experiments of adapting available evaluation methods such as ROUGE and PYRAMID to non-English data. We base our experiments on various English and non-English homogeneous benchmark data sets as well as a non-English heterogeneous data set. Our results indicate that ROUGE can indeed be adapted to non-English data – both homogeneous and heterogeneous. Using a recent implementation of performing an automatic PYRAMID evaluation, we also show its adaptablilty to non-English data. Keywords: summarization evaluation, ROUGE, PYRAMID, automatic evaluation 1. Introduction Research in automatic summarization has long focused on homogeneous, English data, i.e. in the context of the Doc- ument Understanding Conferences (DUC) and Text Analy- sis Conferences (TAC), which resulted in a range of bench- mark data sets.1 In recent years this has expanded to in- clude non-English data (e.g. the MultiLing Shared Task2 as well as heterogeneous sources (e.g. hMDS and auto- hMDS (Zopf et al., 2016b; Zopf, 2018) or even both (e.g. DBS (Benikova et al., 2016). But evaluation methods are still very strongly focused on the English language. ROUGE (Lin, 2004) for example is based on WordNet and an English stop word list. Using the vast parameter space ROUGE offers, the language specific settings can be switched off or moved offline (Mieskes and Pad´o, 2019). Similarly, metrics such as the Jensen-Shannon-Divergence or Kullback-Leibler Divergence, which have been used for the evaluation of summarization (Nenkova and Louis, 2008; Zopf et al., 2016b), can be more easily adapted to the target language if semantic resources are available. The PYRAMID method (Nenkova and Passonneau, 2004) was originally developed as a manual evaluation method. Stud- ies show though that if the whole process is automated, the correlation to human judgments drops considerably, there- fore keeping the need for high-effort manual annotations (Peyrard and Eckle-Kohler, 2017). Recently, Gao et al. (2019) presented a method for calcu- lating PYRAMID scores using embeddings called PyrEval, reducing the need for task-specific resources and enabling content-based evaluation of summaries. Considering the easier availability of embeddings for a given language, this would allow for evaluating summaries across a range of languages independent of time-consuming manual annota- tions. In our work, we present experiments using embeddings- based PYRAMID evaluations on non-English and hetero- geneous data as well as language-independent ROUGE- scores. We aim at answering various research questions gov1https://duc.nist.gov/,https://tac.nist. 2http://multiling.iit.demokritos.gr (Giannakopoulos, 2013) with regards to using existing evaluation metrics such as ROUGE and PYRAMID on this data. Our contributions are therefore as follows: • A new benchmark data set of manual PYRAMID an- notations …

Content truncated to 500 words