Linguistic Corpus Annotation for Automatic Text Simplification Evaluation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1842 - 1866 December 7-11, 2022 ©2022 Association for Computational Linguistics Linguistic Corpus Annotation for Automatic Text Simplification Evaluation Rémi Cardon, Adrien Bibal, Rodrigo Wilkens, David Alfter, Magali Norré, Adeline Müller, Patrick Watrin, Thomas François CENTAL, IL&C, University of Louvain, Belgium {remi.cardon,adrien.bibal,rodrigo.wilkens,david.alfter magali.norre,adeline.muller,patrick.watrin,thomas.francois} @uclouvain.be Abstract Evaluating automatic text simplification (ATS) systems is a difficult task that is either per- formed by automatic metrics or user-based eval- uations. However, from a linguistic point-of- view, it is not always clear on what bases these evaluations operate. In this paper, we propose annotations of the ASSET corpus that can be used to shed more light on ATS evaluation. In addition to contributing with this resource, we show how it can be used to analyze SARI’s be- havior and to re-evaluate existing ATS systems. We present our insights as a step to improve ATS evaluation protocols in the future. 1 Introduction Automatic text simplification (ATS) is a task that consists in automatically adapting a text to make it more accessible for readers. It is the focus of more and more attention from researchers, with a growing number of publications every year and some surveys published recently (Saggion, 2017; Al-Thanyyan and Azmi, 2021; Štajner, 2021). As it is the case in many domains, ATS tasks are cur- rently mostly explored using deep learning (Nisioi et al., 2017; Alva-Manchego et al., 2020b; Cooper and Shardlow, 2020), although rule-based systems are still being explored as well (Saggion, 2017; Evans and Orasan, 2019; Wilkens et al., 2020). One serious hurdle that the field is currently facing is how to evaluate those systems (Grabar and Saggion, 2022). There are two common ap- proaches: human judgment and automatic evalua- tion metrics. Human judgment is collected by ask- ing people to rate the output of a system on a Likert scale using three criteria: grammaticality, meaning preservation and simplicity. This method is cer- tainly encompassing, but requires lots of time and effort. In contrast, automatic evaluation metrics represent a reproducible and quick way to measure the performance of ATS systems. The most com- mon metrics currently used are: BLEU (Papineni et al., 2002), a metric created to evaluate the perfor- mance of machine translation systems; SARI (Xu et al., 2016a), a metric specifically designed for ATS; and the Flesch-Kincaid readability formula (Kincaid et al., 1975). Although these metrics have serious shortcomings (Sulem et al., 2018a; Alva- Manchego et al., 2021; Tanprasert and Kauchak, 2021), their use is widespread in the field due to their ease of use. BLEU and SARI require refer- ence data (human made simplifications) and are known to be more reliable when more references are available. In other words, their behavior is a function of the dataset used as the reference. Since different audiences have different needs when it comes to simplification (Rennes et al., 2022), mak- ing sure that a dataset will reflect those specific needs is important. Two other automatic metrics have been …

Content truncated to 500 words