Supervised and Unsupervised Neural Approaches to Text Readability

Supervised and Unsupervised Neural Approaches to Text Readability Matej Martinc Joˇzef Stefan Institute, Ljubljana, Slovenia Joˇzef Stefan International Postgraduate School, Ljubljana, Slovenia matej.martinc@ijs.si Senja Pollak Joˇzef Stefan Institute, Ljubljana, Slovenia senja.pollak@ijs.si Marko Robnik-ˇSikonja University of Ljubljana, Faculty of Computer and Information Science, Ljubljana, Slovenia marko.robnik@fri.uni-lj.si We present a set of novel neural supervised and unsupervised approaches for determining the readability of documents. In the unsupervised setting, we leverage neural language models, whereas in the supervised setting, three different neural classification architectures are tested. We show that the proposed neural unsupervised approach is robust, transferable across languages, and allows adaptation to a specific readability task and data set. By systematic comparison of several neural architectures on a number of benchmark and new labeled readability data sets in two languages, this study also offers a comprehensive analysis of different neural approaches to readability classification. We expose their strengths and weaknesses, compare their performance to current state-of-the-art classification approaches to readability, which in most cases still rely on extensive feature engineering, and propose possibilities for improvements. 1. Introduction Readability is concerned with the relation between a given text and the cognitive load of a reader to comprehend it. This complex relation is influenced by many factors, such as a degree of lexical and syntactic sophistication, discourse cohesion, and background knowledge (Crossley et al. 2017). In order to simplify the problem of measuring read- ability, traditional readability formulas focused only on lexical and syntactic features Submission received: 26 July 2019; revised version received: 22 November 2020; accepted for publication: 18 December 2020. https://doi.org/10.1162/COLI a 00398 © 2021 Association for Computational Linguistics Published under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license Computational Linguistics Volume 47, Number 1 expressed with statistical measurements, such as word length, sentence length, and word difficulty (Davison and Kantor 1982). These approaches have been criticized because of their reductionism and weak statistical bases (Crossley et al. 2017). Another problem is their objectivity and cultural transferability, since children from different en- vironments master different concepts at different ages. For example, a word television is quite long and contains many syllables but is well-known to most young children who live in families with a television. With the development of novel natural language processing (NLP) techniques, several studies attempted to eliminate deficiencies of traditional readability formulas. These attempts include leveraging high-level textual features for readability modeling, such as semantic and discursive properties of texts. Among them, cohesion and co- herence received the most attention, and several readability predictors based on these text features have been proposed (see Section 2). Nevertheless, none of them seems to predict the readability of the text as well as much simpler readability formulas men- tioned above (Todirascu et al. 2016). With the improvements in machine learning, the focus shifted once again, and most newer approaches consider readability as being a classification, regression, or a ranking task. Machine learning approaches build prediction models to predict human assigned readability scores based on several attributes and manually built features …

Content truncated to 500 words