Constructing Open Cloze Tests Using Generation and Discrimination Capabilities of Transformers

Findings of the Association for Computational Linguistics: ACL 2022, pages 1263 - 1273 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Constructing Open Cloze Tests Using Generation and Discrimination Capabilities of Transformers Mariano Felice, Shiva Taslimipoor and Paula Buttery ALTA Institute, Computer Laboratory, University of Cambridge Cambridge, UK {mf501,st797,pjb48}@cam.ac.uk Abstract This paper presents the first multi-objective transformer model for constructing open cloze tests that exploits generation and discrimina- tion capabilities to improve performance. Our model is further enhanced by tweaking its loss function and applying a post-processing re- ranking algorithm that improves overall test structure. Experiments using automatic and human evaluation show that our approach can achieve up to 82% accuracy according to ex- perts, outperforming previous work and base- lines. We also release a collection of high- quality open cloze tests along with sample sys- tem output and human annotations that can serve as a future benchmark. 1 Introduction Open cloze (Taylor, 1953) tests are a common type of exercise where words are removed from a piece of text and must then be filled in by the students without any options to choose from. They are often used in language learning environments as a quick and effective way to test vocabulary, grammar and reading comprehension (Tremblay, 2011; Trace, 2020). However, designing high-quality cloze tests for language learning is a laborious process that involves finding an optimal distribution of gaps based on aspects such as function, distance, number of answers, etc. (ALTE, 2005; 2011). In this paper, we propose a strategy to con- struct open cloze exercises using transformer mod- els (Vaswani et al., 2017). Our transformer-based architecture employs two objectives to predict the words that should be gapped in a text passage. Our main objective is standard token classification, where we aim to minimise the error of classifying a token as a gap or not. The second and auxil- iary objective is a language-model-based objec- tive whereby we attempt to minimise the language model error when predicting the right answer for each gap. Our solution is based on a pre-trained ELECTRA (Clark et al., 2020) model that is fine- tuned on the two described objectives in a multi- task scenario. Our output aims to mimic the style of open cloze tests in the First Certificate in English (FCE) exam1, which is targeted at learners of English at the B2 proficiency level of the Common European Frame- work of Reference (CEFR) for languages (Council of Europe, 2001). Unlike other tests, the FCE open cloze task aims to simultaneously test many aspects of grammar and vocabulary that students are ex- pected to know at this level. Since the tests are created from a text passage, they must be skilfully designed in order to ensure an optimal distribution of gaps that adheres to guidelines. A shortened example is shown in Figure 1. Our system is evaluated under two settings: 1) automatic evaluation, where the generated gaps are compared to gold-standard gaps proposed by test experts, and 2) human evaluation, where the quality …

Content truncated to 500 words