Cloze Quality Estimation for Language Assessment

Findings of the Association for Computational Linguistics: EACL 2023, pages 540–550 May 2-6, 2023 ©2023 Association for Computational Linguistics Cloze Quality Estimation for Language Assessment Zizheng Zhang † and Masato Mita ‡† and Mamoru Komachi† †Tokyo Metropolitan University 6-6 Asahigaoka, Hino, Tokyo 191-0065, Japan ‡CyberAgent, Inc. 2-24-12 Shibuya Shibuya-ku, Tokyo 150-6121, Japan zhang-zizheng@ed.tmu.ac.jp, mita_masato@cyberagent.co.jp, komachi@tmu.ac.jp Abstract Cloze tests play an essential role in language assessment and help language learners improve their skills. In this paper, we propose a novel task called Cloze Quality Estimation (CQE) — a zero-shot task of evaluating whether a cloze test is of sufficient “high-quality” for language assessment based on two important factors: re- liability and validity. We have taken the first step by creating a new dataset named CELA for the CQE task, which includes English cloze tests and corresponding evaluations about their quality annotated by native English speakers, which includes 2,597 and 1,730 instances in aspects of reliability and validity, respectively. We have tested baseline evaluation methods on the dataset, showing that our method could contribute to the CQE task, but the task is still challenging. 1 1 Introduction A cloze test (Taylor, 1953) is an efficient and comprehensive tool in language assessment, and thus it is widely used in language proficiency tests (Passage and Questions in Table 1), which mea- sure multiple language abilities of examinees simul- taneously, for example, grammatical knowledge (Rye, 1982; Alderson, 1979) and reading com- prehension ability (Raymond, 1988; Klein-Braley, 1997). The most widespread format of the cloze test is the multiple-choice word-level cloze test, which consists of an incomplete passage with sev- eral blanks and a series of questions, where each question includes several options (four options in the usual setting), requiring examinees to fill the blanks by selecting words (or phrases) from options that make the passage coherent. The significantly high price of creating cloze tests by experts has prompted automatic cloze gen- eration methods (Goto et al., 2010; Sakaguchi 1The CELA dataset and code of baselines are available at https://github.com/zz-zhang/ cloze-quality-estimation. et al., 2013; Hill and Simha, 2016; Panda et al., 2022). However, automatically generated cloze tests do not always contribute well to language assessment and suffer from low quality. In lan- guage assessment, a reliable and valid cloze test shows a high ability to measure examinees’ lan- guage level (Bachman, 1985). Cloze question cre- ation has two steps, word deletion and distractor generation, with the latter greatly affecting the abil- ity to measure language level. As indicated by (Xie et al., 2018), some cloze tests, particularly automat- ically generated cloze tests, are created coarsely and cause two fatal issues in language assessment: (1) these tests do not guarantee that the answer is not ambiguous, which means there is a risk that multiple options fit almost equally well into the blank; (2) the test creation process considers less about which aspect of the language phenomenon is measured in the test; hence, such tests cannot measure the examinee’s language level. These two issues make the cloze test …

Content truncated to 500 words