A Neural Pairwise Ranking Model for Readability Assessment

Findings of the Association for Computational Linguistics: ACL 2022, pages 3802 - 3813 May 22-27, 2022 c⃝2022 Association for Computational Linguistics A Neural Pairwise Ranking Model for Readability Assessment Justin Lee∗ University of Toronto CSA Group Toronto, Canada chunhin.lee@mail.utoronto.ca Sowmya Vajjala National Research Council Ottawa, Canada sowmya.vajjala@nrc-cnrc.gc.ca Abstract Automatic Readability Assessment (ARA), the task of assigning a reading level to a text, is traditionally treated as a classification problem in NLP research. In this paper, we propose the first neural, pairwise ranking approach to ARA and compare it with existing classification, re- gression, and (non-neural) ranking methods. We establish the performance of our model by conducting experiments with three English, one French and one Spanish datasets. We demonstrate that our approach performs well in monolingual single/cross corpus testing sce- narios and achieves a zero-shot cross-lingual ranking accuracy of over 80% for both French and Spanish when trained on English data. Ad- ditionally, we also release a new parallel bilin- gual readability dataset in English and French. To our knowledge, this paper proposes the first neural pairwise ranking model for ARA, and shows the first results of cross-lingual, zero- shot evaluation of ARA with neural models. 1 Introduction Automatic Readability Assessment is the task of assigning a reading level for a given text. It is useful in many applications from selecting age ap- propriate texts in classrooms (Sheehan et al., 2014) to assessment of patient education materials (Sare et al., 2020) and clinical informed consent forms (Perni et al., 2019). Contemporary NLP approaches treat it primarily as a classification problem which makes it non-transferable to situations where the reading level scale is different from the model. Ap- plying learning to rank methods has been seen as a potential solution to this problem in the past. Rank- ing texts by readability is also useful in a range of application scenarios, from ranking search re- sults based on readability (Kim et al., 2012; Four- ney et al., 2018) to controlling the reading level of machine translation output (Agrawal and Carpuat, ∗Work done during an internship at National Research Council, Canada 2019; Marchisio et al., 2019). However, explo- ration of ranking methods has not been a promi- nent direction for ARA research. Further, recent developments in neural ranking approaches haven’t been explored for this task yet, to our knowledge. ARA typically relies on the presence of large amounts of data labeled by reading level. Further, although linguistic features are common in ARA research, it is challenging to calculate them for sev- eral languages, due to lack of available software support. Though there is a lot of recent interest in neural network based cross-lingual transfer learn- ing approaches for various NLP tasks, there hasn’t been much research in this direction for ARA yet. With this context, we address two research ques- tions in this paper: 1. Is neural, pairwise ranking a better approach than classification or regression for ARA? 2. Is zero-shot, cross-lingual transfer possible for ARA models through ranking? The main contributions of this …

Content truncated to 500 words