CEFR-Based Sentence Difficulty Annotation and Assessment

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6206 - 6219 December 7-11, 2022 ©2022 Association for Computational Linguistics CEFR-Based Sentence Difficulty Annotation and Assessment Yuki Arase† and Satoru Uchida⋆ and Tomoyuki Kajiwara⋄ †Graduate School of Information Science and Technology, Osaka University, Japan ⋆Faculty of Languages and Cultures, Kyushu University, Japan ⋄Graduate School of Science and Engineering, Ehime University, Japan arase@ist.osaka-u.ac.jp, uchida@flc.kyushu-u.ac.jp kajiwara@cs.ehime-u.ac.jp Abstract Controllable text simplification is a crucial as- sistive technique for language learning and teaching. One of the primary factors hinder- ing its advancement is the lack of a corpus an- notated with sentence difficulty levels based on language ability descriptions. To address this problem, we created the CEFR-based Sen- tence Profile (CEFR-SP) corpus, containing 17k English sentences annotated with the lev- els based on the Common European Frame- work of Reference for Languages assigned by English-education professionals. In addition, we propose a sentence-level assessment model to handle unbalanced level distribution because the most basic and highly proficient sentences are naturally scarce. In the experiments in this study, our method achieved a macro-F1 score of 84.5% in the level assessment, thus outper- forming strong baselines employed in readabil- ity assessment. 1 Introduction Controllable text simplification, first proposed by Scarton and Specia (2018), is the automatic rewrit- ing of sentences to make them comprehensible to a target audience with a specific proficiency level. Among its primary applications are providing read- ing assistance to language learners and helping teachers adjust the difficulty level of their teaching materials (Petersen and Ostendorf, 2007; Pellow and Eskenazi, 2014; Paetzold, 2016). The fine- grained control of output levels to match the lin- guistic ability of the readership is crucial for these educational applications. While readability assessments have been actively studied (e.g., in (Vajjala Balakrishna, 2015; Meng et al., 2020; Deutsch et al., 2020)), linking read- ability to language ability is difficult. Readabil- ity scores, such as the Flesch–Kincaid grade level (Kincaid et al., 1975), are intended for native speak- ers, not for language learners to whom very differ- ent considerations apply. Pilán et al. (2014) and Ozasa et al. (2007) revealed that readability met- rics designed for L1 do not apply to L2 learners. Furthermore, readability definitions use documents rather than sentences, which are required by text simplification at the sentence-level, as their unit. The lack of a corpus annotated by sentence dif- ficulty level hinders the advancement of control- lable text simplification. Previous studies (Scarton and Specia, 2018; Nishihara et al., 2019; Agrawal et al., 2021) necessarily used corpora annotated for readability rather than difficulty; furthermore, they assumed that all sentences in a document had the same readability (i.e., the document level in Newsela (Xu et al., 2015)). To solve these problems, we created a large-scale English corpus annotated by sentence difficulty lev- els based on the Common European Framework of Reference for Languages (CEFR),1 the most widely used international standard describing learners’ lan- guage ability. Our CEFR-based Sentence Profile (CEFR-SP) corpus adapts CEFR to sentence levels. …

Content truncated to 500 words