Trends, Limitations and Open Challenges in Automatic Readability Assessment Research

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5366–5377 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5366 Trends, Limitations and Open Challenges in Automatic Readability Assessment Research Sowmya Vajjala National Research Council, Canada sowmya.vajjala@nrc-cnrc.gc.ca Abstract Readability assessment is the task of evaluating the reading difficulty of a given piece of text. This article takes a closer look at contemporary NLP research on developing computational models for readability assessment, identifying the common approaches used for this task, their shortcomings, and some challenges for the future. Where possible, the survey also connects computational research with insights from related work in other disciplines such as education and psychology. Keywords: readability assessment, survey, resources and evaluation 1. Introduction Automatic Readability Assessment (ARA) refers to the task of modeling the reading and comprehension difficulty of a given piece of text, for a given tar- get audience. This has a broad range of applications in both machine facing and human facing scenarios. Some examples of human facing scenarios are: choos- ing appropriate reading materials for language teach- ing (Collins-Thompson and Callan, 2004), supporting readers with learning disabilities (Rello et al., 2012) and self-directed learning (Beinborn et al., 2012). In machine facing scenarios, ARA is used in scenarios such as for ranking search results by their reading level (Kim et al., 2012), generating translated text controlled for reading level (Marchisio et al., 2019; Agrawal and Carpuat, 2019), and evaluating automatic text simpli- fication (Alva-Manchego et al., 2020). TextEvaluator TM1, used to determine whether a reading material is ap- propriate for a grade level in classroom instruction, is a well known real world application of ARA. Apart from such uses around and within the field of NLP, the gen- eral idea of readability assessment is used in a range of other scenarios. A common case is of medical research, where it was used for assessing patient education ma- terials (Sare et al., 2020) and consent forms (Perni et al., 2019; Lyatoshinsky et al., 2019), for example. This broad application range highlights ARA as one of the important applications of NLP. Research into measuring how difficult (or easy) is a text to read is now a century old (e.g., Thorndike (1921), Lively and Pressey (1923), Vogel and Wash- burne (1928)). Early research focused on creating lists of difficult words and/or developing a readability “formula”, which is a simple weighted linear function of easy to calculate variables such as number/length of words/sentences in a text, percentage of difficult words etc. This resulted in several readability formulas such as Flesch Reading Ease (Flesch, 1948), SMOG (McLaughlin, 1969), Dale-Chall readability formula 1https://textevaluator.ets.org/ TextEvaluator/ (Dale and Chall, 1948) etc. (see Dubay (2007) for a detailed survey of such formulae). NLP researchers started taking interest in this problem only in the past two decades. From statistical language models and feature engineering based machine learn- ing approaches to more recent deep neural networks, a range of approaches have been explored so far for this …

Content truncated to 500 words