A Unified Neural Network Model for Readability Assessment with Feature Projection and Length-Balanced Loss

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7446 - 7457 December 7-11, 2022 ©2022 Association for Computational Linguistics A Unified Neural Network Model for Readability Assessment with Feature Projection and Length-Balanced Loss Wenbiao Li1,2, Ziyang Wang1,2, Yunfang Wu1,3∗ 1MOE Key Laboratory of Computational Linguistics, Peking University 2School of Software and Microelectronics, Peking University, Beijing, China 3School of Computer Science, Peking University, Beijing, China {liwb, wzy232303}@stu.pku.edu.cn, wuyf@pku.edu.cn Abstract For readability assessment, traditional meth- ods mainly employ machine learning classifiers with hundreds of linguistic features. Although the deep learning model has become the promi- nent approach for almost all NLP tasks, it is less explored for readability assessment. In this pa- per, we propose a BERT-based model with fea- ture projection and length-balanced loss (BERT- FP-LBL) for readability assessment. Specially, we present a new difficulty knowledge guided semi-supervised method to extract topic fea- tures to complement the traditional linguistic features. From the linguistic features, we em- ploy projection filtering to extract orthogonal features to supplement BERT representations. Furthermore, we design a new length-balanced loss to handle the greatly varying length distri- bution of data. Our model achieves state-of-the- art performances on two English benchmark datasets and one dataset of Chinese textbooks, and also achieves the near-perfect accuracy of 99% on one English dataset. Moreover, our pro- posed model obtains comparable results with human experts in consistency test. 1 Introduction Readability assessment is to automatically deter- mine the difficulty level of a given text, aim- ing to recommend suitable reading materials to readers. There are wide applications of readabil- ity assessment, such as automating readers’ advi- sory (Pera and Ng, 2014), clinical informed consent forms (Perni et al., 2019) and internet-based patient education materials (Sare et al., 2020). Comparing with other natural language pro- cessing (NLP) tasks, readability assessment is less explored. In the early days, researchers exploit linguistic features to develop various readability formulas, such as Flesch (Flesch, 1948), Dale-Chall (Dale and Chall, 1948) and ∗Corresponding author. SMOG (Mc Laughlin, 1969). Later, the main- stream research (Deutsch et al., 2020; Hansen et al., 2021; Lee et al., 2021) is to employ machine learn- ing models to classify a text, by designing a large number of linguistic features. There are also works that treat it as a regression task (Sheehan et al., 2010) or a ranking task (Lee and Vajjala, 2022). Recently, unlike other NLP tasks, the introduc- tion of deep neural networks for readability assess- ment does not achieve overwhelming advantages over traditional machine learning methods. Em- ploying neural network models for readability as- sessment, there are several challenges: (1) The scale of the dataset for readability assess- ment is small, which restricts the performance of deep neural network models. (2) The deep neural network model is mainly based on characters or words and the extracted fea- tures are often at a shallow level. As a result, words with similar functions or meanings, such as “man” and “gentleman”, are mapped into close vectors al- though their …

Content truncated to 500 words