CU : Computational Assessment of Short Free Text Answers - A Tool for Evaluating Students’ Understanding

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 603–607, Atlanta, Georgia, June 14-15, 2013. c⃝2013 Association for Computational Linguistics CU : Computational Assessment of Short Free Text Answers - A Tool for Evaluating Students’ Understanding Ifeyinwa Okoye Institute of Cognitive Science Dept. of Computer Science University of Colorado Boulder, CO 80309, USA okoye@colorado.edu Steven Bethard Institute of Cognitive Science Dept. of Computer Science University of Colorado Boulder, CO 80309, USA bethard@colorado.edu Tamara Sumner Institute of Cognitive Science Dept. of Computer Science University of Colorado Boulder, CO 80309, USA sumner@colorado.edu Abstract Assessing student understanding by evaluat- ing their free text answers to posed questions is a very important task. However, manually, it is time-consuming and computationally, it is difficult. This paper details our shallow NLP approach to computationally assessing student free text answers when a reference answer is provided. For four out of the five test sets, our system achieved an overall accuracy above the median and mean. 1 Introduction Assessing student understanding is one of the holy grails of education (Redecker et al., 2012). If we (teachers, tutors, intelligent tutors, potential employ- ers, parents and school administrators) know what and how much a student knows, then we know what the student still needs to learn. And then, can ef- ficiently and effectively educate the student. How- ever, the task of assessing what exactly a student un- derstands about a particular topic can be expensive, difficult and subjective. Using multiple choice questionnaires is one of the most prevalent forms of assessing student under- standing because it is easy and fast, both manually and computationally. However there has been a lot of pushback from educators about the validity of re- sults gotten from multiple choice questionnaires. Assessing student understanding by evaluating student free text answers either written or spoken is one of the preferred alternatives to multiple choice questionnaires. As an assessment tool, free text an- swers can illuminate what and how much a student knows since the student is forced to recall terms and make connections between those terms rather than just picking one out of several options. However, assessing free text answers manually is tedious, ex- pensive and time-consuming, hence the search for a computational option. There are three main issues that can limit the com- putational approach and corresponding performance when assessing free text answers: (1) the unit of assessment, (2) the reference and (3) the level of assessment. The unit of assessment can be words, facets, phrases, sentences, short answers or essays. The reference is the correct answer and what is being compared to the student answer. Most re- searchers generate the reference manually (Noorbe- hbahani and Kardan, 2011; Graesser et al., 2004) but some have focused on automatically generating the reference (Ahmad, 2009). The level of assessment can be coarse with 2 categories such as correct and incorrect or more finer-grained with up to 19 cate- gories as in (Ahmad, 2009). In general, the finer- …

Content truncated to 500 words