Open-ended Knowledge Tracing for Computer Science Education

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3849 - 3862 December 7-11, 2022 ©2022 Association for Computational Linguistics Open-Ended Knowledge Tracing for Computer Science Education Naiming Liu∗, Zichao Wang∗, Richard G. Baraniuk Rice University nl35,zw16,richb@rice.edu Andrew Lan University of Massachusetts Amherst andrewlan@cs.umass.edu Abstract In education applications, knowledge tracing refers to the problem of estimating students’ time-varying concept/skill mastery level from their past responses to questions and predicting their future performance. One key limitation of most existing knowledge tracing methods is that they treat student responses to questions as binary-valued, i.e., whether they are cor- rect or incorrect. Response correctness analy- sis/prediction ignores important information on student knowledge contained in the exact con- tent of the responses, especially for open-ended questions. In this paper, we conduct the first exploration into open-ended knowledge tracing (OKT) by studying the new task of predicting students’ exact open-ended responses to ques- tions. Our work is grounded in the domain of computer science education with programming questions. We develop an initial solution to the OKT problem, a student knowledge-guided code generation approach, that combines pro- gram synthesis methods using language models with student knowledge tracing methods. We also conduct a series of quantitative and quali- tative experiments on a real-world student code dataset to validate OKT and demonstrate its promise in educational applications. 1 Introduction Knowledge tracing (KT) (Corbett and Anderson, 1994) refers to the problem of estimating student mastery of concepts/skills/knowledge components from their responses to questions and using these estimates to predict their future performance. KT methods play a key role in many of today’s large- scale online learning platforms to automatically estimate the knowledge levels of a large number of students and provide each of them with person- alized feedback and recommendation, leading to improved learning outcomes (Ritter et al., 2007). KT methods consist of two essential components. *The first two authors contributed equally. First, a knowledge estimation (KE) component, i.e., ht+1 = KE((p1, x1), . . . , (pm, xt)), (1) estimates a student’s current knowledge state ht+1 using questions (p) and responses (x) from previ- ous (discrete) time steps for this student. Second, a response prediction (RP) component predicts the student’s response to the next question (or future questions), i.e., xt+1 ∼ RP(ht+1, pt+1). Section 4 contains a detailed overview of existing KT meth- ods and how the question, responses, and knowl- edge state variables are represented. One key limitation of almost all existing KT methods is that they only analyze and predict binary-valued student responses to questions, i.e., the correctness of the response. That is, the RP is typically a simple binary classifier. As a result, one can broadly apply KT methods to any question as long as student responses are graded. However, this approach loses important information regarding stu- dent mastery, since it does not make use of the con- tent of questions and student responses, especially for open-ended questions. Past work has shown that students’ open-ended responses to such ques- tions contain …

Content truncated to 500 words