Semantic Structural Evaluation for Text Simplification

Proceedings of NAACL-HLT 2018, pages 685–696 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Semantic Structural Evaluation for Text Simplification Elior Sulem, Omri Abend, Ari Rappoport Department of Computer Science, The Hebrew University of Jerusalem {eliors|oabend|arir}@cs.huji.ac.il Abstract Current measures for evaluating text simplifi- cation systems focus on evaluating lexical text aspects, neglecting its structural aspects. In this paper we propose the first measure to ad- dress structural aspects of text simplification, called SAMSA. It leverages recent advances in semantic parsing to assess simplification qual- ity by decomposing the input based on its se- mantic structure and comparing it to the out- put. SAMSA provides a reference-less auto- matic evaluation procedure, avoiding the prob- lems that reference-based methods face due to the vast space of valid simplifications for a given sentence. Our human evaluation experi- ments show both SAMSA’s substantial corre- lation with human judgments, as well as the deficiency of existing reference-based mea- sures in evaluating structural simplification.1 1 Introduction Text simplification (TS) addresses the translation of an input sentence into one or more simpler sen- tences. It is a useful preprocessing step for several NLP tasks, such as machine translation (Chan- drasekar et al., 1996; Mishra et al., 2014) and rela- tion extraction (Niklaus et al., 2016), and has also been shown useful in the development of reading aids, e.g., for people with dyslexia (Rello et al., 2013) or non-native speakers (Siddharthan, 2002). The task has attracted much attention in the past decade (Zhu et al., 2010; Woodsend and La- pata, 2011; Wubben et al., 2012; Siddharthan and Angrosh, 2014; Narayan and Gardent, 2014), but has yet to converge on an evaluation protocol that yields comparable results across different methods and strongly correlates with human judgments. This is in part due to the difficulty to combine the effects of different simplification operations 1All data and code are available in https://github. com/eliorsulem/SAMSA. (e.g., deletion, splitting and substitution). Xu et al. (2016) has recently made considerable progress towards that goal, and proposed to tackle it both by using an improved reference-based measure, named SARI, and by increasing the number of ref- erences. However, their research focused on lex- ical, rather than structural simplification, which provides a complementary view of TS quality as this paper will show. This paper focuses on the evaluation of the structural aspects of the task. We introduce the semantic measure SAMSA (Simplification Auto- matic evaluation Measure through Semantic An- notation), the first structure-aware measure for TS in general, and the first to use semantic structure in this context in particular. SAMSA stipulates that an optimal split of the input is one where each predicate-argument structure is assigned its own sentence, and measures to what extent this asser- tion holds for the input-output pair in question, by using semantic structure. SAMSA focuses on the core semantic components of the sentence, and is tolerant towards the deletion of other units.2 For example, SAMSA will assign a high score to the output split “John got …

Content truncated to 500 words