CS1QA: A Dataset for Assisting Code-based Question Answering in an Introductory Programming Course

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2026 - 2040 July 10-15, 2022 ©2022 Association for Computational Linguistics CS1QA: A Dataset for Assisting Code-based Question Answering in an Introductory Programming Course Changyoon Lee, Yeon Seonwoo, Alice Oh School of Computing, KAIST changyoon.lee@kaist.ac.kr, yeon.seonwoo@kaist.ac.kr alice.oh@kaist.edu Abstract We introduce CS1QA, a dataset for code-based question answering in the programming ed- ucation domain. CS1QA consists of 9,237 question-answer pairs gathered from chat logs in an introductory programming class using Python, and 17,698 unannotated chat data with code1. Each question is accompanied with the student’s code, and the portion of the code relevant to answering the question. We carefully design the annotation process to construct CS1QA, and analyze the collected dataset in detail. The tasks for CS1QA are to predict the question type, the relevant code snip- pet given the question and the code and retriev- ing an answer from the annotated corpus. Re- sults for the experiments on several baseline models are reported and thoroughly analyzed. The tasks for CS1QA challenge models to un- derstand both the code and natural language. This unique dataset can be used as a benchmark for source code comprehension and question answering in the educational setting. 1 Introduction Question answering (QA) studies systems that understand questions and the relevant context to provide answers. Question forms include single document QA (Rajpurkar et al., 2016), multi-hop QA (Yang et al., 2018), conversational QA (Reddy et al., 2019), and open domain QA (Kwiatkowski et al., 2019). Questions about specific domains are asked in NewsQA (Trischler et al., 2016) and TechQA (Castelli et al., 2020), and images are pro- vided with the question in visual QA (Antol et al., 2015). Another interesting field of QA asks ques- tions about source code (Liu and Wan, 2021). A useful application of QA is in the educational domain. Asking questions and getting the answer is an essential and efficient means of learning. In this paper, we focus on QA for programming education, 1The code and the data used in this paper can be found at https://github.com/cyoon47/CS1QA. Figure 1: An example of our data tuple. Each data tuple consists of {question, answer, question type, code, rele- vant code lines}. We annotate the type of each question and the code lines (orange) relevant to the question. where both the input modes and the domain pose interesting challenges. Answering these questions requires reading and understanding both source code and natural language questions. In addition, students’ questions are often complex, demanding thorough understanding of the context such as the intention and the educational goal to answer them. Recently, models that understand programming languages (PL) have been studied, and show promising results in diverse code comprehension tasks (Alon et al., 2018; Feng et al., 2020; Guo et al., 2021). However, these models have limi- tations to support question answering. They are not trained on datasets containing questions about the code and are not designed for QA tasks. …

Content truncated to 500 words