Guiding the Growth: Difficulty-Controllable Question Generation through Step-by-Step Rewriting

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5968–5978 August 1–6, 2021. ©2021 Association for Computational Linguistics 5968 Guiding the Growth: Difficulty-Controllable Question Generation through Step-by-Step Rewriting Yi Cheng1, Siyao Li2, Bang Liu3∗, Ruihui Zhao1, Sujian Li4, Chenghua Lin5, Yefeng Zheng1 1Tencent Jarvis Lab, China 2LTI, Carnegie Mellon University 3RALI & Mila, Université de Montréal 4Peking University 5The University of Sheffield {yicheng, zacharyzhao, yefengzheng}@tencent.com, siyaol@andrew.cmu.edu, bang.liu@umontreal.ca, lisujian@pku.edu.cn, c.lin@shef.ac.uk Abstract This paper explores the task of Difficulty- Controllable Question Generation (DCQG), which aims at generating questions with re- quired difficulty levels. Previous research on this task mainly defines the difficulty of a ques- tion as whether it can be correctly answered by a Question Answering (QA) system, lack- ing interpretability and controllability. In our work, we redefine question difficulty as the number of inference steps required to answer it and argue that Question Generation (QG) systems should have stronger control over the logic of generated questions. To this end, we propose a novel framework that progressively increases question difficulty through step-by- step rewriting under the guidance of an ex- tracted reasoning chain. A dataset is automat- ically constructed to facilitate the research, on which extensive experiments are conducted to test the performance of our method. 1 Introduction The task of Difficulty-Controllable Question Gen- eration (DCQG) aims at generating questions with required difficulty levels and has recently attracted researchers’ attention due to its wide application, such as facilitating certain curriculum-learning- based methods for QA systems (Sachan and Xing, 2016) and designing exams of various difficulty levels for educational purpose (Kurdi et al., 2020). Compared to previous QG works which control the interrogative word (Zi et al., 2019; Kang et al., 2019) or the context of a question (Liu et al., 2020, 2019a), few works have been conducted on diffi- culty control, as it is hard to formally define the difficulty of a question. To the best of our knowl- edge, Gao et al. (2019) is the only previous work of DCQG for free text, and defines question difficulty as whether a QA model can correctly answer it. ∗Corresponding author. Figure 1: An example of generating a complex ques- tion through step-by-step rewriting based on the reason- ing chain extracted from a constructed context graph. This definition gives only two difficulty levels and is mainly empirically driven, lacking interpretabil- ity for what difficulty is and how difficulty varies. In this work, we redefine the difficulty level of a question as the number of inference steps required to answer it, which reflects the requirements on reasoning and cognitive abilities (Pan et al., 2019). Existing QA systems perform substantially worse in answering multi-hop questions than single-hop ones (Yang et al., 2018), also supporting the sound- ness of using reasoning hops to define difficulty. To achieve DCQG with the above definition, a QG model should have strong control over the logic and reasoning complexity of generated questions. Graph-based methods are well …

Content truncated to 500 words