Investigating Text Simplification Evaluation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 876–882 August 1–6, 2021. ©2021 Association for Computational Linguistics 876 Investigating Text Simplification Evaluation Laura V´asquez-Rodr´ıguez1, Matthew Shardlow2, Piotr Przybyła3, Sophia Ananiadou1 1National Centre for Text Mining, The University of Manchester, Manchester, United Kingdom 2Department of Computing and Mathematics, Manchester Metropolitan University, Manchester, United Kingdom 3Institute of Computer Science, Polish Academy of Sciences, Warsaw, Poland {laura.vasquezrodriguez, sophia.ananiadou}@manchester.ac.uk m.shardlow@mmu.ac.uk piotr.przybyla@ipipan.waw.pl Abstract Modern text simplification (TS) heavily relies on the availability of gold standard data to build machine learning models. However, ex- isting studies show that parallel TS corpora contain inaccurate simplifications and incor- rect alignments. Additionally, evaluation is usually performed by using metrics such as BLEU or SARI to compare system output to the gold standard. A major limitation is that these metrics do not match human judgements and the performance on different datasets and linguistic phenomena vary greatly. Further- more, our research shows that the test and training subsets of parallel datasets differ sig- nificantly. In this work, we investigate existing TS corpora, providing new insights that will motivate the improvement of existing state-of- the-art TS evaluation methods. Our contribu- tions include the analysis of TS corpora based on existing modifications used for simplifica- tion and an empirical study on TS models per- formance by using better-distributed datasets. We demonstrate that by improving the distribu- tion of TS datasets, we can build more robust TS models. 1 Introduction Text Simplification transforms natural language from a complex to a simple format, with the aim to not only reach wider audiences (Rello et al., 2013; De Belder and Moens, 2010; Aluisio et al., 2010; Inui et al., 2003) but also as a preprocessing step in related tasks (Shardlow, 2014; Silveira and Branco, 2012). Simplifications are achieved by using parallel datasets to train sequence-to-sequence text gen- eration algorithms (Nisioi et al., 2017) to make complex sentences easier to understand. They are typically produced by crowdsourcing (Xu et al., 2016; Alva-Manchego et al., 2020a) or by align- ment (Cao et al., 2020; Jiang et al., 2020). They are infamously noisy and models trained on these give poor results when evaluated by humans (Cooper and Shardlow, 2020). In this paper we add to the growing narrative around the evaluation of natu- ral language generation (van der Lee et al., 2019; Caglayan et al., 2020; Pang, 2019), focusing on parallel text simplification datasets and how they can be improved. Why do we need to re-evaluate TS resources? In the last decade, TS research has relied on Wikipedia-based datasets (Zhang and Lapata, 2017; Xu et al., 2016; Jiang et al., 2020), despite their known limitations (Xu et al., 2015; Alva-Manchego et al., 2020a) such as questionable sentence pairs alignments, inaccurate simplifications and a limited variety of simplification modifications. Apart from affecting the reliability of models trained on these datasets, their low quality influences the evaluation relying on automatic metrics that requires gold- standard simplifications, such as SARI (Xu et al., 2016) and BLEU (Papineni et al., 2001). …

Content truncated to 500 words