CEPOC: The Cambridge Exams Publishing Open Cloze dataset

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4285–4290 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4285 CEPOC: The Cambridge Exams Publishing Open Cloze dataset Mariano Felice, Shiva Taslimipoor, Øistein E. Andersen, Paula Buttery ALTA Institute, Computer Laboratory, University of Cambridge Cambridge, UK {mf501, st797, oa223, pjb48}@cam.ac.uk Abstract Open cloze tests are a standard type of exercise where examinees must complete a text by filling in the gaps without any given options to choose from. This paper presents the Cambridge Exams Publishing Open Cloze (CEPOC) dataset, a collection of open cloze tests from world-renowned English language proficiency examinations. The tests in CEPOC have been expertly designed and validated using standard principles in language research and assessment. They are prepared for language learners at different proficiency levels and hence classified into different CEFR levels (A2, B1, B2, C1, C2). This resource can be a valuable testbed for various NLP tasks. We perform a complete set of experiments on three tasks: gap filling, gap prediction, and CEFR text classification. We implement transformer-based systems based on pre-trained language models to model each task and use our dataset as a test set, providing promising benchmark results. Keywords: open cloze, blank-filling, language learning, second language testing, Cambridge examinations 1. Introduction The cloze test (Taylor, 1953) is a standard testing pro- cedure where certain words are replaced with gaps in a piece of text, which must then be filled by the student. There are different variations of the original cloze test: Open cloze No options are given to fill in the blank. Students must produce the answer with no help. Multiple choice An answer must be chosen from a set of given options containing the key (i.e. correct answer) and distractors. C-test A modified version of the open cloze test where the first few letters of the answer are kept in the gap as a hint (Raatz and Klein-Braley, 1981). An example of each cloze type is given in Figure 1. Cloze tests are widely used for testing language pro- ficiency due to their simplicity and efficiency (Fotos, 1991; Jonz, 1991; Tremblay, 2011; Trace, 2020). How- ever, they must be carefully designed in order to pro- vide accurate measures of ability, often requiring cal- ibration using methods such as those from Item Re- sponse Theory (IRT) (Lord, 1980). Although there is vast research into modelling multiple-choice cloze tests (mainly in the field of read- ing comprehension, e.g. Xie et al. (2018) or Kurdi et al. (2020)), to the best of our knowledge, public datasets of calibrated cloze tests are non-existent, hindering re- search into automated testing. In this paper, we attempt to address this problem by making the following con- tributions: 1) we release the first dataset of expertly designed and calibrated open cloze tests at different proficiency levels in English, 2) we provide benchmark results for three tasks: gap generation, gap filling and full-text proficiency level prediction, and 3) we offer insights into how …

Content truncated to 500 words