Robustness of Hybrid Models in Cross-domain Readability Assessment

Robustness of Hybrid Models in Cross-domain Readability Assessment Ho Hung Lim, Tianyuan Cai, John S. Y. Lee, Meichun Liu Department of Linguistics and Translation City University of Hong Kong {hhlim3,jsylee,meichliu}@cityu.edu.hk, tianycai-c@my.cityu.edu.hk Abstract Recent studies in automatic readability assess- ment have shown that hybrid models — mod- els that leverage both linguistically motivated features and neural models — can outperform neural models. However, most evaluations on hybrid models have been based on in-domain data in English. This paper provides further evidence on the contribution of linguistic fea- tures by reporting the first direct comparison between hybrid, neural and linguistic models on cross-domain data. In experiments on a Chinese dataset, the hybrid model outperforms the neural model on both in-domain and cross- domain data. Importantly, the hybrid model ex- hibits much smaller performance degradation in the cross-domain setting, suggesting that the linguistic features are more robust and can bet- ter capture salient indicators of text difficulty. 1 Introduction Automatic Readability Assessment (ARA) predicts how difficult it is for the reader to understand a text. Traditional machine learning approaches for ARA typically train statistical classifiers with hand- crafted features (Pitler and Nenkova, 2008; Sung et al., 2015). Similar to other NLP tasks, neural approaches have recently achieved superior perfor- mance (Tseng et al., 2019; Azpiazu and Pera, 2019; Martinc et al., 2021). Combining linguistic fea- tures and neural models has been found to benefit a variety of NLP tasks (Lei et al., 2018; Strubell et al., 2018). While these ‘hybrid’ models have also been applied in ARA, there have been varying re- sults ranging from no effect (Deutsch et al., 2020), marginal improvement (Filighera et al., 2019), to significant improvement (Lee et al., 2021). Past studies comparing hybrid and neural mod- els have mostly been conducted in an in-domain setting, with the training and test data drawn from the same source. However, real-word applications of ARA models are often targeted at cross-domain or cross-corpus data. Consider the task of retriev- ing extra-curricular reading materials for language learning from web texts, which likely diverge in style and content from the training data. In-domain evaluation therefore may not accurately reflect the actual performance on such tasks. This paper focuses on the task of predicting the grade level of an input text. We present the first comparison between hybrid, neural and lin- guistic models on this task in the cross-domain setting. Our contribution is two-fold. First, we show that the hybrid model outperforms the neural model both in-domain and cross-domain in Chinese datasets, providing further evidence on the contri- bution of linguistic features. Second, the hybrid model exhibits much smaller performance degrada- tion on cross-domain data, suggesting their robust- ness and ability to capture more salient indicators of text difficulty.1 After a review of previous work (Section 2), we present our datasets (Section 3). We then outline our approach (Section 4) and report experimental results (Section 5). 2 Background 2.1 Hybrid model design Statistical classifiers can be trained on a variety of features, capturing lexical, syntactic …

Content truncated to 500 words