Automatic Distractor Generation for Multiple Choice Questions in Standard Tests

Proceedings of the 28th International Conference on Computational Linguistics, pages 2096–2106 Barcelona, Spain (Online), December 8-13, 2020 2096 Automatic Distractor Generation for Multiple Choice Questions in Standard Tests Zhaopeng Qiu Tencent Medical AI Lab zhaopengqiu@tencent.com Xian Wu ∗ Tencent Medical AI Lab kevinxwu@tencent.com Wei Fan Tencent Medical AI Lab davidwfan@tencent.com Abstract To assess the knowledge proficiency of a learner, multiple choice question is an efficient and widespread form in standard tests. However, the composition of the multiple choice question, especially the construction of distractors is quite challenging. The distractors are required to both incorrect and plausible enough to confuse the learners who did not master the knowledge. Currently, the distractors are generated by domain experts which are both expensive and time- consuming. This urges the emergence of automatic distractor generation, which can benefit various standard tests in a wide range of domains. In this paper, we propose a quEstion and an- swer guided Distractor GEneration (EDGE) framework to automate distractor generation. EDGE consists of three major modules: (1) the Reforming Question Module and the Reforming Passage Module apply gate layers to guarantee the inherent incorrectness of the generated distractors; (2) the Distractor Generator Module applies attention mechanism to control the level of plausibil- ity. Experimental results on a large-scale public dataset demonstrate that our model significantly outperforms existing models and achieves a new state-of-the-art. 1 Introduction Standard test, such as TOFEL and SAT, is an efficient and essential tool to assess knowledge proficiency of a learner (Ch and Saha, 2018). According to testing results, teachers or ITS (Intelligent Tutoring System) services can develop personalized study plans for different students. When organizing a standard test, a vital issue is to select a suitable question form. Among various question forms, multiple choice question (MCQ) is widely adopted in many notable tests, such as GRE, TOFEL and SAT. MCQs have many advantages including less testing time, more objective and easy on the grader (Ch and Saha, 2018). A typical MCQ consists of a stem and several candidate answers, among which one is correct, the rest are distractors. As shown in Figure 1, in addition to a stem, some tests also include a long reading passage to provide the context of this MCQ. The quality of an MCQ depends heavily on the quality of the distractors. If the distractors can not confuse students, the correct answer could be concluded easily. As a result, the discrimination of the question will degrade, and the test will also lose the ability of the assessment. However, it is a challenging job to design useful and qualified distractors. Rather than being a trivial wrong answer, the distractor should have the plausibility which confuses learners who did not master the knowledge (Liang et al., 2018; Qiu et al., 2019). A good distractor should be grammatically correct given the question and semantically consistent with the passage context of the question (Gao et al., 2018). Meanwhile, the question composers need to enhance the plausibility of the distractor without hurting its inherent incorrectness. …

Content truncated to 500 words