Gauging the Gap Between Human and Machine Text Simplification Through Analytical Evaluation of Simplification Strategies and Errors

Findings of the Association for Computational Linguistics: EACL 2023, pages 359–375 May 2-6, 2023 ©2023 Association for Computational Linguistics Gauging the Gap Between Human and Machine Text Simplification Through Analytical Evaluation of Simplification Strategies and Errors Daichi Yamaguchi Rei Miyata Sayuka Shimada Satoshi Sato Nagoya University, Nagoya, Japan {yamaguchi.daichi.e4,shimada.sayuka.y9}@s.mail.nagoya-u.ac.jp {miyata,ssato}@nuee.nagoya-u.ac.jp Abstract This study presents an analytical evaluation of neural text simplification (TS) systems. Be- cause recent TS models are trained in an end-to- end fashion, it is difficult to grasp their abilities to perform particular simplification operations. For the advancement of TS research and devel- opment, we should understand in detail what current TS systems can and cannot perform in comparison with human performance. To that end, we first developed an analytical eval- uation framework consisting of fine-grained taxonomies of simplification strategies (at both the surface and content levels) and errors. Us- ing this framework, we annotated TS instances produced by professional human editors and multiple neural TS systems and compared the results. Our analyses concretely and quantita- tively revealed a wide gap between humans and systems, specifically indicating that systems tend to perform deletions and local substitu- tions while excessively omitting important in- formation, and that the systems can hardly per- form information addition operations. Based on our analyses, we also provide detailed direc- tions to address these limitations. 1 Introduction Text simplification (TS) is the task of reducing the content and structural complexity of text while re- taining the core part of the original meaning (Alva- Manchego et al., 2020). TS can not only facilitate the text reading by children or language learners, but also improve the performance of downstream NLP applications, including machine translation and summarization (Siddharthan et al., 2004; Šta- jner and Popovic, 2016). Early studies on TS have separately dealt with lexical simplification (Glavaš and Štajner, 2015) and syntactic simplification (Scarton et al., 2017), and developed simplification techniques special- ized for particular linguistic phenomena. In con- trast, recent studies have tackled TS as a task of Aarti has a growth mindset. Simplified by system Aarti has a growth mindset, the belief that your intelligence can grow. Aarti has a growth mindset, which means he believes that your intelligence can grow. Surface Strategy: Content Strategy: Error: Complex Simplified by human Grammaticality: 5, Meaning preservation: 4, Simplicity: 5 Delete at phrase level Delete detail / extra information No error Grammaticality: 5, Meaning preservation: 5, Simplicity: 4 Surface Strategy: Content Strategy: Error: Replace at clause level Paraphrase into an explanatory expression No error Figure 1: Example of an analytical evaluation in terms of editing strategies and errors. monolingual translation from a complex to a sim- plified language using deep neural networks. While neural TS has demonstrated steady improvement, few studies have attempted to assess what kind of editing operations are performed by the systems in concrete terms. To further advance TS research and development, we should understand the potential and limitations of current TS technologies and pre- cisely grasp the gap between human and machine TS. To do …

Content truncated to 500 words