Readability and Linearity

Readability and Linearity Yo Ehara Tokyo Gakugei University Faculty of Education, 4-1-1 Nukuikita-machi, Koganei-shi, Tokyo 184-8501 Japan ehara@u-gakugei.ac.jp Abstract This paper investigates automatic assessments of text readability for second language learn- ers. The goodness of text readability as- sessment methods has been evaluated using the correlation coefficients between their read- ability scores and labels annotated to the texts by language teachers. For measuring this cor- relation, previous papers have used Pearson’s correlation coefficient (ρ). In this paper, we show that the use of Pearson’s ρ is inappro- priate through experiments because it is eas- ily affected by the linearity of the labels and scores, which are often not linear. In contrast, the linearity has no influence on rank corre- lations, which should be used for evaluating readability assessors. We also propose an un- supervised readability assessor that takes word difficulty for second language learners into ac- count and achieves the best among all unsu- pervised assessors in our experiments in terms of Pearson’s ρ and the rank correlations. 1 Introduction Assessing readability plays an essential role in sec- ond language acquisition; it can be used for many educational applications such as intelligent reading support systems and placement tests for language classes. Readability assessment is a costly task for educational experts and language teachers. To per- form it, they must read a text and assess its read- ability by guessing how difficult the text is for tar- get learning readers. Hence, to reduce the cost of the labor required by educational experts, the task of automatically identifying the readability of texts for language learners, known as automatic readabil- ity assessment (ARA), has been extensively studied in the field of artificial intelligence (AI). Unsupervised automatic readability assessment appeared early but has recently been reexamined as a research focus. Early studies such as the Dale-Chall formula (1948) (Dale and Chall(1948)), the Flesch Reading Ease formula (Flesch(1948)) (1948), and the Flesch-Kincaid readability formula (1975) (Kin- caid et al.(1975)Kincaid, Fishburne Jr, Rogers, and Chissom) were unsupervised, as they did not use costly annotated readability labels. Given a text, these formulae calculate its readability score based on simple superficial textual features such as the average length of a word in the given text. How- ever, most of these early formulae are designed to assess readability for children who are native speakers. Evaluation datasets with readability la- bels annotated by language teachers targeting sec- ond language learners appeared much later in the 2010s (Feng et al.(2010)Feng, Jansche, Huener- fauth, and Elhadad; Xia et al.(2016)Xia, Kochmar, and Briscoe; Vajjala and Luˇci´c(2018)). In these works, automatic readability assessment tasks using these evaluation datasets were formalized as a su- pervised document classification problem, and sub- stantial research efforts were invested into the con- struction of classifiers by feature engineering to find complicated textual features that correlate well with readability labels. Recently, Martinc et al.(2021)Martinc, Pollak, and Robnik-ˇSikonja) revisited the unsupervised ap- proach. They proposed that the perplexity scores of neural language models can also be used to rep- Figure 1: …

Content truncated to 500 words