Entropy as a Proxy for Gap Complexity in Open Cloze Tests

Proceedings of Recent Advances in Natural Language Processing, pages 323–327, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_037 323 Entropy as a Proxy for Gap Complexity in Open Cloze Tests Mariano Felice Paula Buttery ALTA Institute Computer Laboratory University of Cambridge Cambridge, UK {mf501,pjb48}@cam.ac.uk Abstract This paper presents a pilot study of entropy as a measure of gap complexity in open cloze tests aimed at learners of English. Entropy is used to quantify the information content in each gap, which can be used to estimate com- plexity. Our study shows that average gap entropy correlates positively with proficiency levels while individual gap entropy can cap- ture contextual complexity. To the best of our knowledge, this is the first unsupervised information-theoretical approach to evaluating the quality of cloze tests. 1 Introduction Fill-in-the-gap or cloze test exercises are common means of assessing grammar and vocabulary in the realm of English as a Foreign Language (EFL). The most common example is the multiple choice question, which presents the student with a gapped sentence and a set of possible answers from which the right one is to be selected. These are referred to as closed cloze questions, since the answer is limited to the alternatives given. On the contrary, open cloze questions do not provide predefined options, so the student must produce an answer from scratch. Generating these exercises is a laborious process, since they must be carefully designed to ensure they test the desired learning objective and do not confuse or present trivial questions to the student. For this reason, choosing the optimal locations in a sentence to insert the gaps and defining a suitable set of answer options becomes crucial, especially when exercises are generated automatically. In this paper, we focus on open cloze tests and show how entropy can be used to assess the com- plexity of each gap in the text. Entropy is shown to provide insights into the expected difficulty of the question and correlate directly with the target pro- ficiency level of the exercises. Exploiting this in- formation should thus facilitate the automatic gen- eration of more reliable open cloze exercises. 2 Related Work Work on automated cloze test generation has mostly focused on multiple choice questions and distractor selection (Mitkov and Ha, 2003; Sumita et al., 2005; Brown et al., 2005; Lee and Seneff, 2007; Lin et al., 2007; Smith et al., 2010; Sak- aguchi et al., 2013). Conversely, there has been little work on open cloze tests. Pino et al. (2008) describes a strategy to generate open cloze ques- tions using example sentences from a learners’ dictionary. Sentences are chosen based on four lin- guistic criteria: (grammatical) complexity, well- defined context (collocations), grammaticality and length. Further work improved on this method by providing hints for the gapped words (Pino and Es- kenazi, 2009). Malafeev (2014) developed an open source sys- tem to emulate open cloze tests in Cambridge En- glish exams based on the most frequent gapped words. Expert EFL instructors found the gen- erated gaps to be …

Content truncated to 500 words