Learning Syntactic Dense Embedding with Correlation Graph for Automatic Readability Assessment

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3013–3025 August 1–6, 2021. ©2021 Association for Computational Linguistics 3013 Abstract Deep learning models for automatic readability assessment generally discard linguistic features traditionally used in machine learning models for the task. We propose to incorporate linguistic features into neural network models by learning syntactic dense embeddings based on linguistic features. To cope with the relationships between the features, we form a correlation graph among features and use it to learn their embeddings so that similar features will be represented by similar embeddings. Experiments with six data sets of two proficiency levels demonstrate that our proposed methodology can complement BERT-only model to achieve significantly better performances for automatic readability assessment. 1 Introduction Readability is the ease with which a reader can understand a written text1. Predicting readability has been widely applied in education (Lennon and Burdick, 2004), book publishing (Pera and Ng, 2014), marketing (Chebat et al., 2003), newspaper readership (Pitler and Nenkova, 2008), and health information communication (Bernstam et al., 2005). Ever since the first study by Lively and Pressey in 1923, many researchers have developed various popular readability formulas including Flesch (Flesch, 1948), Fog (Gunning, 1969) and Lexile (Stenner et al., 1988). These traditional readability formulas are favored by domain applications due to their simplicity even though the formulas are mostly based on shallow features and known to lack accuracy (Bruce et al., 1981; Davison and Kantor, 1982; Graesser et al., 2004). * Corresponding authors. Its strong reliance on expert knowledge is also a burden to adapt it to a new domain. Machine learning approaches, which incorporate a broader set of morphological, lexical, syntactic, and discourse features, have shown to achieve better accuracy in readability assessment (Si and Callan, 2001; Collins-Thompson and Callan, 2005). Figure 1 (a) describes a generic machine-learning framework for Automatic Readability Assessment (ARA) where manual feature engineering is an important step to extract important linguistic features for building readability classification models. To bypass the necessity of heavy feature engineering, deep learning strategies have been studied to automatically detect patterns or extract features related to readability (Azpiazu and Pera, 2019; Martinc et al., 2019; Mohammadi and Khasteh, 2019). Figure 1 (b) provides a generic neural network structure of deep learning approach to ARA. While neural network models take word embedding as input, they in general discard linguistic features traditionally used in machine learning models (Deutsch et al., 2020). If ever incorporated, linguistic features such as POS and morphological tags are only used to guide attention mechanism for embedding representation of the text (Azpiazu and Pera, 2019). Pre-trained models such as BERT (Devlin et al., 2019) learn dense representations of text by informing the models with semantically neighboring words, sentences, or context. Despite the attempts of recent research to assess BERT’s ability to implicitly capture the structural properties of language (Goldberg, 2019; Jawahar et al., 2019; Kovaleva et al., 2019), it has been observed …

Content truncated to 500 words