Evaluating the Readability of Text Simplification Output for Readers with Cognitive Disabilities

Evaluating the Readability of Text Simplification Output for Readers with Cognitive Disabilities Victoria Yaneva*, Irina Temnikova**, Ruslan Mitkov* *Research Institute in Information and Language Processing,University of Wolverhampton, **Qatar Computing Research Institute,HBKU, Doha, Qatar, *Wolverhampton,UK, **Doha,Qatar, V.Yaneva@wlv.ac.uk, itemnikova@qf.org.qa, R.Mitkov@wlv.ac.uk Abstract This paper presents an approach for automatic evaluation of the readability of text simplification output for readers with cognitive disabilities. First, we present our work towards the development of the EasyRead corpus, which contains easy-to-read documents created especially for people with cognitive disabilities. We then compare the EasyRead corpus to the simplified output contained in the LocalNews corpus (Feng, 2009), the accessibility of which has been evaluated through reading comprehension experiments including 20 adults with mild intellectual disability. This comparison is made on the basis of 13 disability-specific linguistic features. The comparison reveals that there are no major differences between the two corpora, which shows that the EasyRead corpus is to a similar reading level as the user-evaluated texts. We also discuss the role of Simple Wikipedia (Zhu et al., 2010) as a widely-used accessibility benchmark, in light of our finding that it is significantly more complex than both the EasyRead and the LocalNews corpora. Keywords: readability, text simplification, cognitive disabilities, autism, easy-to-read 1. Introduction The main task of automatic text simplification (TS) is to convert texts into a more understandable form for readers with lower than average reading skills, without changing the original meaning of the text. These readers could in- clude children and people with poor literacy (Aluisio et al., 2010), learning disabilities (Feng, 2009), neurodevel- opmental disorders (Rello et al., 2013), second language learners (Petersen, 2007), among others. The evaluation of TS output could be defined through the tasks of quality and accessibility (readability) estimation. Quality estimation is performed at sentence level and its purpose is to establish whether the output sentences are grammatical, meaningful and simpler. Readability estima- tion on the other hand is done at text level and is concerned with measuring how understandable the output is for the target reader population. In this paper we focus solely on discussing readability evaluation. Currently, readability of TS output is measured in two ways: automatic evaluation through readability formulae and human evaluation, where sample texts are evaluated through reading comprehension experiments involving tar- get readers. While the latter is certainly the most reliable way to measure text accessibility and harness insights into user preferences, it is very time-consuming and expensive to perform, especially in cases where the target readers are people with cognitive disabilities (intellectual disability, autism, hyperactivity, etc.). To address this issue, research has been focusing on developing automatic methods for readability evaluation, which could account for the specific reading difficulties of various reader populations (Dubay, 2004; Benjamin, 2012), including people with cognitive disabilities (Feng et al., 2010; Yaneva and Evans, 2015). However, relying on readability formulae to measure text accessibility is an approach that has many drawbacks (Ben- jamin, 2012; Dubay, 2004; Siddharthan, 2004), mainly re- lated to the fact that readability formulae only employ sur- …

Content truncated to 500 words