ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformations

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4668–4679 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4668 ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformations Fernando Alva-Manchego1∗and Louis Martin2,3∗and Antoine Bordes3 Carolina Scarton1 and Benoˆıt Sagot2 and Lucia Specia1,4 1University of Sheffield, 2Inria, 3Facebook AI Research, 4Imperial College London f.alva@sheffield.ac.uk, louismartin@fb.com, abordes@fb.com c.scarton@sheffield.ac.uk, benoit.sagot@inria.fr l.specia@imperial.ac.uk Abstract In order to simplify a sentence, human ed- itors perform multiple rewriting transforma- tions: they split it into several shorter sen- tences, paraphrase words (i.e. replacing com- plex words or phrases by simpler synonyms), reorder components, and/or delete information deemed unnecessary. Despite these varied range of possible text alterations, current mod- els for automatic sentence simplification are evaluated using datasets that are focused on a single transformation, such as lexical para- phrasing or splitting. This makes it impossi- ble to understand the ability of simplification models in more realistic settings. To alleviate this limitation, this paper introduces ASSET, a new dataset for assessing sentence simplifi- cation in English. ASSET is a crowdsourced multi-reference corpus where each simplifica- tion was produced by executing several rewrit- ing transformations. Through quantitative and qualitative experiments, we show that simpli- fications in ASSET are better at capturing characteristics of simplicity when compared to other standard evaluation datasets for the task. Furthermore, we motivate the need for devel- oping better methods for automatic evaluation using ASSET, since we show that current pop- ular metrics may not be suitable when multiple simplification transformations are performed. 1 Introduction Sentence Simplification (SS) consists in modifying the content and structure of a sentence to make it easier to understand, while retaining its main idea and most of its original meaning (Alva-Manchego et al., 2020). Simplified texts can benefit non-native speakers (Paetzold, 2016), people suffering from aphasia (Carroll et al., 1998), dyslexia (Rello et al., 2013) or autism (Evans et al., 2014). They also help language processing tasks, such as parsing (Chan- drasekar et al., 1996), summarisation (Silveira and ∗Equal Contribution Branco, 2012), and machine translation (Hasler et al., 2017). In order simplify a sentence, several rewriting transformations can be performed: replacing com- plex words/phrases with simpler synonyms (i.e. lex- ical paraphrasing), changing the syntactic structure of the sentence (e.g. splitting), or removing super- fluous information that make the sentence more complicated (Petersen, 2007; Alu´ısio et al., 2008; Bott and Saggion, 2011). However, models for automatic SS are evaluated on datasets whose sim- plifications are not representative of this variety of transformations. For instance, TurkCorpus (Xu et al., 2016), a standard dataset for assessment in SS, contains simplifications produced mostly by lexical paraphrasing, while reference simplifica- tions in HSplit (Sulem et al., 2018a) focus on split- ting sentences. The Newsela corpus (Xu et al., 2015) contains simplifications produced by pro- fessionals applying multiple rewriting transforma- tions, but sentence alignments are automatically computed and thus imperfect, and its data can only be accessed after signing a restrictive public- sharing licence and …

Content truncated to 500 words