Does Summary Evaluation Survive Translation to Other Languages?

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2425 - 2435 July 10-15, 2022 ©2022 Association for Computational Linguistics Does Summary Evaluation Survive Translation to Other Languages? Spencer Braun1, Oleg Vasilyev1, Neslihan Iskender2, John Bohannon1 1Primer Technologies Inc., San Francisco, California {spencer.braun,oleg,john}@primer.ai 2Technische Universität Berlin, Quality and Usability Lab neslihan.iskender@tu-berlin.de Abstract The creation of a quality summarization dataset is an expensive, time-consuming effort, requir- ing the production and evaluation of summaries by both trained humans and machines. The returns to such an effort would increase sig- nificantly if the dataset could be used in ad- ditional languages without repeating human annotations. To investigate how much we can trust machine translation of summariza- tion datasets, we translate the English Sum- mEval dataset to seven languages and com- pare performances across automatic evaluation measures. We explore equivalence testing as the appropriate statistical paradigm for eval- uating correlations between human and auto- mated scoring of summaries. We also con- sider the effect of translation on the relative performance between measures. We find some potential for dataset reuse in languages simi- lar to the source and along particular dimen- sions of summary quality. Our code and data can be found at https://github.com/ PrimerAI/primer-research/. 1 Introduction A large summarization dataset includes thousands of texts and human-written summaries (for exam- ple, CNN/Daily Mail (Hermann et al., 2015)). In order to make it applicable for wider research, it may also contain machine-generated summaries by many models, accompanied by human and ma- chine evaluations of the quality of the generated summaries (Fabbri et al., 2021). The human an- notation alone is a complicated effort, requiring careful planning and setup (Kryscinski et al., 2020; Tang et al., 2021; Iskender et al., 2021). What purpose do the human annotations serve? Their main utility is serving as a benchmark for automated evaluation measures. Researchers de- sign measures to closely approximate human judg- ment in order to increase the pace of summarization model improvement. As summarization resources grow for English-language models, it becomes in- creasingly important to consider whether we can repurpose these datasets for use in other languages as well. Given a method that could produce flawless translations, the original human annotations quite clearly remain useful, as the relative rankings of the summaries would be invariant. In this sce- nario, comparing automated measures in another language with the English human scores produces valid conclusions. In practice, however, translation will introduce some distortions — both mild and extreme — that can spoil the utility of the original annotations. While a "uniform" distortion over all texts would preserve the relations among evaluation measures, this too is an unrealistic assumption as translation will correct and simplify some texts, introduce er- rors into others, and push components of text qual- ity like relevance, coherence, and fluency in dif- ferent directions (Fomicheva et al., 2021; Freitag et al., 2021). We are left to ask how to determine whether it is still practical to rely …

Content truncated to 500 words