Automatic Generation of Socratic Subquestions for Teaching Math Word Problems

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4136 - 4149 December 7-11, 2022 ©2022 Association for Computational Linguistics Automatic Generation of Socratic Subquestions for Teaching Math Word Problems Kumar Shridhar ∗♠ Jakub Macina ∗ ♠Φ Mennatallah El-Assady ♠Φ Tanmay Sinha ▼ Manu Kapur ▼ Mrinmaya Sachan ♠ ♠Department of Computer Science, ETH Zurich ΦETH AI Center ▼Professorship for Learning Sciences and Higher Education, ETH Zurich Abstract Socratic questioning is an educational method that allows students to discover answers to complex problems by asking them a series of thoughtful questions. Generation of didacti- cally sound questions is challenging, requir- ing understanding of the reasoning process in- volved in the problem. We hypothesize that such questioning strategy can not only enhance the human performance, but also assist the math word problem (MWP) solvers. In this work, we explore the ability of large language models (LMs) in generating sequential ques- tions for guiding math word problem-solving. We propose various guided question generation schemes based on input conditioning and rein- forcement learning. On both automatic and hu- man quality evaluations, we find that LMs con- strained with desirable question properties gen- erate superior questions and improve the over- all performance of a math word problem solver. We conduct a preliminary user study to exam- ine the potential value of such question genera- tion models in the education domain. Results suggest that the difficulty level of problems plays an important role in determining whether questioning improves or hinders human perfor- mance. We discuss the future of using such questioning strategies in education. https://github.com/eth-nlped/ scaffolding-generation 1 Introduction Questioning can be a valuable way of supporting student thinking. It can be conceived as a scaffold (Wood et al., 1976; Quintana et al., 2004), where a more knowledgeable tutor helps a student in solv- ing problems otherwise too difficult. One approach well-suited for mathematics is funneling (Wood, 1994), which uses prompting questions to guide students towards a solution. ∗ Equal contribution; correspondence at: {shkumar, macinaj}@ethz.ch Goal-driven Focused How many eggs does Janet sell? Is duck an animal? How many eggs does each duck lay? How much does Janet make at the farmers' market? Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market? Figure 1: Math word problems can be precedurally solved in multiple reasoning steps. One operationaliza- tion of Socratic questioning is to map each step in the procedure to a question. Asking (machines/humans) the right set of questions in a certain sequence (shown in green) can be an effective way to do so. In order to be effective, the Socratic questioning should be focused and goal-driven. Figure 1 shows an example of a math word problem where this questioning strategy might be beneficial. We hypothesize that these questions …

Content truncated to 500 words