Prompt-based Learning for Text Readability Assessment

Findings of the Association for Computational Linguistics: EACL 2023, pages 1819–1824 May 2-6, 2023 ©2023 Association for Computational Linguistics Prompt-based Learning for Text Readability Assessment Bruce W. Lee1,2,‡, Jason Hyung-Jong Lee2 1University of Pennsylvania - PA, USA 2LXPER AI Research - Seoul, South Korea brucelws@seas.upenn.edu jasonlee@lxper.com Abstract We propose the novel adaptation of a pre- trained seq2seq model for readability assess- ment. We prove that a seq2seq model – T5 or BART – can be adapted to discern which text is more difficult from two given texts (pairwise). As an exploratory study to prompt-learn a neu- ral network for text readability in a text-to-text manner, we report useful tips for future work in seq2seq training and ranking-based approach to readability assessment. Specifically, we test nine input-output formats/prefixes and show that they can significantly influence the final model performance. Also, we argue that the combination of text-to-text training and pairwise ranking setup 1) enables leveraging multiple parallel text simplification data for teaching readability and 2) trains a neural model for the general concept of readability (therefore, better cross-domain generalization). At last, we report a 99.6% pairwise classification accuracy on Newsela and a 98.7% for OneStopEnglish, through a joint training approach. Our code is available at github.com/brucewlee/prompt-lea rning-readability. 1 Introduction Readability assessment evaluates the reading diffi- culty of a given piece of text (Vajjala, 2021). The early traditional readability assessment methods like Flesch-Kincaid Grade Level (Kincaid et al., 1975) utilized a linear regression formula fitted to data from large-scale reading experiments on human subjects. More recently, readability assess- ment has often been viewed as a classification task (Feng et al., 2010). Under this classification-based task formulation, models using various handcrafted features (Xia et al., 2016; Vajjala and Meurers, 2012), computer-generated features (Martinc et al., 2021; Imperial, 2021), or both (Lee et al., 2021) ‡Woong Sung (Bruce) Lee was on leave from the Univer- sity of Pennsylvania during the research period. have been reported. Showing the potential that neural modeling is more suitable than handcrafted features in holistically capturing the inherent lin- guistic properties that affect readability. Among the varying approaches to readability assessment, fine-tuning deep transformer models (Vaswani et al., 2017), that are pre-trained with lan- guage modeling objectives (e.g. Masked Language Modelling, Next Sentence Prediction), has proven highly effective in multiple reports (Lee and Vaj- jala, 2022; Lee et al., 2021). So far, encoder-only transformer architectures like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have been the go-to approach, and few reports discuss the applicability of other architecture types. Further, there is no report on how readability assessment can be cast in a text-to-text task formulation (§2). But recent reports (Raffel et al., 2020) show that a text-to-text is promising for multiple downstream tasks. The main contribution of this paper is that we fine-tune full encoder-decoder transformer archi- tectures (also referred to as sequence-to-sequence) and check if they can learn about text readability. A sequence-to-sequence model has been previously fine-tuned for downstream tasks like document ranking (Nogueira …

Content truncated to 500 words