A BERT-based Distractor Generation Scheme with Multi-tasking and Negative Answer Training Strategies.

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4390–4400 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4390 A BERT-based Distractor Generation Scheme with Multi-tasking and Negative Answer Training Strategies Ho-Lam Chung1, Ying-Hong Chan2, Yao-Chung Fan3 Department of Computer Science and Engineering National Chung Hsing University, Taichung, Taiwan 1holam.chung@protonmail.com 2harry831120@gmail.com 3yfan@nchu.edu.tw Abstract In this paper, we investigate the following two limitations for the existing distractor genera- tion (DG) methods. First, the quality of the existing DG methods are still far from prac- tical use. There are still room for DG qual- ity improvement. Second, the existing DG de- signs are mainly for single distractor genera- tion. However, for practical MCQ preparation, multiple distractors are desired. Aiming at these goals, in this paper, we present a new dis- tractor generation scheme with multi-tasking and negative answer training strategies for ef- fectively generating multiple distractors. The experimental results show that (1) our model advances the state-of-the-art result from 28.65 to 39.81 (BLEU 1 score) and (2) the gener- ated multiple distractors are diverse and shows strong distracting power for multiple choice question. 1 Introduction Given a passage, a question, and an answer phrase, the goal of distractor generation (DG) is to gener- ate context-related wrong options (i.e., distractor) for multiple-choice questions (MCQ). Pioneering research (Gao et al., 2019; Yeung et al., 2019; Zhou et al., 2019) have demonstrated the feasibility of generating distractors based on deep learning tech- niques. While significant advances for DG were reported in the literature, we find that the existing DG results are still far from practical use. In this paper, we investigate the following two issues for distractor generation: (1) DG quality improvement and (2) Multiple distractor generation. DG Quality Improvement There is still room to be improved for high-quality distractor generation. By manually examining the DG results generated by the existing method, we find that the results are still far from ideal for practical use. Thus, one Example 1 Context Omitted. (See Appendix) Question · Why did Mr.King want to send Henry away? Answer · Because Henry was too lazy. Gen. Distractors ·d1 : Because Henry didn’t want to go. ·d2 : Because Henry didn’t want to go to the bookstore. Example 2 Context Omitted. (See Appendix) Question · Which of the following women would look most attractive? Answer · A short red-haired woman who wears a purple hat. Gen. Distractors ·d1 : A young woman who wears a white hat. ·d2 : A woman who wears a white hat. Table 1: Two examples for showing the issue of gen- erating multiple distractors by a simple beam search: Note that the generated distractors (i.e., d1 and d2) are the same statements with only slight word usage dif- ference. Such results lower the distracting power for MCQ preparation. goal of our research is to improve the DG quality further. For the quality issues, in this paper, we explore BERT model’s employment for performance im- provement. As known, employing transformer- based language models has …

Content truncated to 500 words