Pushing on Text Readability Assessment: A Transformer Meets Handcrafted Linguistic Features

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10669–10686 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10669 Pushing on Text Readability Assessment: A Transformer Meets Handcrafted Linguistic Features Bruce W. Lee1,3 Univ. of Pennsylvania1 PA, USA brucelws@seas.upenn.edu Yoo Sung Jang2,3 Univ. of Wisconsin-Madison2 WI, USA yjang43@wisc.edu Jason Hyung-Jong Lee3 LXPER AI3 Seoul, South Korea jasonlee@lxper.com Abstract We report two essential improvements in read- ability assessment: 1. three novel features in advanced semantics and 2. the timely evidence that traditional ML models (e.g. Random For- est, using handcrafted features) can combine with transformers (e.g. RoBERTa) to augment model performance. First, we explore suitable transformers and traditional ML models. Then, we extract 255 handcrafted linguistic features using self-developed extraction software. Fi- nally, we assemble those to create several hy- brid models, achieving state-of-the-art (SOTA) accuracy on popular datasets in readability as- sessment. The use of handcrafted features help model performance on smaller datasets. Nota- bly, our RoBERTA-RF-T1 hybrid achieves the near-perfect classification accuracy of 99%, a 20.3% increase from the previous SOTA. 1 Introduction The long quest for advancing readability assess- ment (RA) mostly centered on handcrafting the linguistic features that affect readability (Pitler and Nenkova, 2008). RA is a time-honored branch of natural language processing (NLP) that quantifies the difficulty with which a reader understands a text (Feng et al., 2010). Being one of the oldest system- atic approaches to linguistics (Collins-Thompson, 2014), RA developed various linguistic features. These range from simple measures like the aver- age count of syllables to those as sophisticated as semantic complexity (Buchanan et al., 2001). Perhaps due to the abundance of dependable lin- guistic features, an overwhelming majority of RA systems are Support Vector Machines (SVM) with handcrafted features (Hansen et al., 2021). Such traditional machine learning (ML) methods were linguistically explainable, expandable, and most importantly, competent against the modern neural models. As a fragmentary example, Filighera et al. (2019) reports that a large ensemble of 6 BiLSTMs with BERT (Devlin et al., 2019), ELMo (Peters et al., 2018), Word2Vec (Mikolov et al., 2013), and GloVe (Pennington et al., 2014) embeddings showed only ∼1% accuracy improvement from a single SVM model developed by Xia et al. (2016). Even though deep neural networks have achieved state-of-the-art (SOTA) performance in almost all semantic tasks where sufficient data were available (Collobert et al., 2011; Zhang et al., 2015), neural models started showing promising results in RA only quite recently (Martinc et al., 2021). A known challenge for the researchers in RA is the lack of large public datasets – with the unique exception of WeeBit (Vajjala and Meurers, 2012). Technically speaking, even WeeBit is not entirely public since it has to be directly obtained from the authors. Martinc et al. (2021) raised the SOTA classi- fication accuracy on the popular WeeBit dataset (Vajjala and Meurers, 2012) by about 4% using BERT. This was the first solid proof that neural models with auto-generated features can show sig- nificant improvement compared to traditional ML …

Content truncated to 500 words