Difficulty-aware Distractor Generation for Gap-Fill Items

Difficulty-aware Distractor Generation for Gap-Fill Items Chak Yan Yeung, John Lee, Benjamin Tsou Department of Linguistics and Translation City University of Hong Kong cyyeung91@gmail.com, {jsylee,rlbtsou}@cityu.edu.hk Abstract Many computer-assisted language learning (CALL) systems offer gap-fill items, often with multiple choices in order to facilitate im- mediate feedback. Automatic distractor gener- ation can therefore be helpful in providing the multiple choices. While existing algorithms focus on proposing the most plausible distrac- tors, many realistic scenarios make use of dis- tractors at a variety of difficulty levels. This paper evaluates the use of a neural language model to rank distractors in terms of difficulty. Experiments show that BERT outperforms se- mantic similarity measures, in terms of both correlation to human judgment and classifica- tion accuracy of distractor plausibility. 1 Introduction Many computer-assisted language learning (CALL) systems offer gap-fill items, also known as cloze or fill-in-the-blank items. A gap-fill item consists of a carrier sentence in which one word — called the key, or target word — is blanked out. Table 1 shows an example carrier sentence whose target word is ‘served’. To enable automatic feedback, multiple choices are sometimes provided for filling the gap. As shown in Table 1, these choices include the tar- get word itself and several distractors. Judicious selection of distractors is important for generating effective items. A distractor should produce a sen- tence that seems plausible, yet unacceptable. Lan- guage pedagogy literature generally recommends that the target word and distractors belong to the same word class (Heaton, 1989), and be semanti- cally related, ideally “false synonyms” (Goodrich, 1977). An empirical study confirmed that distrac- tors indeed tend to be syntactically and semanti- cally homogenous (Pho et al., 2014). To reduce the manual effort and time needed for selecting distractors, there has been much interest He as class representative for two years. (a) served [target word] (b) acted [distractor] (c) brought [distractor] Table 1: A multiple-choice gap-fill item consists of a carrier sentence with a blank, and choices for filling the blank. In this example, the choices include two distrac- tors and the target word (correct answer). in developing algorithms for automatic distractor generation. Existing algorithms typically take a two-step approach (Jiang and Lee, 2017; Susanti et al., 2018). The first step generates distractor candidates, typically in a list ranked according to measures of semantic similarity and collocation strength. The second step removes candidates that are acceptable answers, using n-gram and colloca- tion frequency or other criteria. Evaluations on distractor generation tend to be limited to the highest-ranked distractors, for example the top-ranked or top three candidates only (Jiang and Lee, 2017; Susanti et al., 2018). Many practical scenarios, however, require not only the most challenging distractors, but distrac- tors across the spectrum of plausibility. When au- thoring test items, for example, it can be useful to generate items at various difficulty levels for com- prehensive assessment. In a CALL system, it can be effective to personalize distractor difficulty ac- cording to the user’s language proficiency. It is informative, then, …

Content truncated to 500 words