Free-text input vs menu selection: exploring the difference with a tutorial dialogue system.

Free-text input vs menu selection: exploring the difference with a tutorial dialogue system. Jenny McDonald1, Alistair Knott2 and Richard Zeng1 1Higher Education Development Centre, University of Otago 2Department of Computer Science, University of Otago jenny.mcdonald@otago.ac.nz Abstract We describe the in-class evaluation of two versions of a tutorial dialogue sys- tem with 338 volunteers from a first-year undergraduate health-sciences class. One version uses supervised machine-learning techniques to classify student free-text re- sponses; the other requires students to se- lect their preferred response from a series of options (menu-based). Our results indi- cate that both the free-text and menu-based tutors produced significant gains on imme- diate post-test scores compared to a control group. In addition, there was no significant difference in performance between students in the free-text and menu-based conditions. We note specific analysis work still to do as part of this research and speculate briefly on the potential for using tutorial dialogue systems in real class settings. 1 Introduction In large undergraduate classes (1500-1800 stu- dents), it is time-consuming, costly and seldom practical to provide students with feedback on their conceptions other than by computer-based marking of formative assessments. Typical exam- ples of this include Learning Management System (LMS) based multiple-choice quizzes or similar. Most computer-assisted assessment involves stu- dents being able to recognise a correct response rather than recall and independently generate an answer. In the context of the first-year undergrad- uate health sciences course that we studied, cur- rently all computer-assisted assessment takes this form. In 2008, the coordinator of a first year un- dergraduate health sciences class asked us about ways in which technologies might assist students to practice writing short-answer questions. As a result of this request, we wanted to investigate whether students answering questions with free- text or multiple-choice(menu-based) selections in a tutorial dialogue setting would result in perfor- mance gains on student test scores and whether there would be any difference in performance be- tween students who generated free-text answers and those who selected their preferred answer from a number of options. In the next section we begin with a brief literature review from the fields of both Education and Cognitive Science. Next, we briefly describe the design and features of our tutorial dialogue system. The experimen- tal design, and results are described in subsequent sections and we conclude with a discussion of our key findings. 2 Background This study is situated at the boundaries between at least three established fields of inquiry: ed- ucational assessment research; psychological re- search, in particular the study of memory, recog- nition and recall; and finally intelligent tutoring systems (ITS) and natural language processing (NLP) research. Since the 1920s the positive benefits on stu- dent performance of answering practice, or for- mative assessment, questions have been demon- strated in classroom studies (Frederiksen, 1984). Similar positive effects have been demonstrated in psychology laboratory studies since the 1970s. (McDaniel et al., 2007) Large meta-analytic ed- ucational studies looking at the impact of prac- tice tests on student outcomes indicate that on …

Content truncated to 500 words