Using the text to evaluate short answers for reading comprehension exercises

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task, pages 286–295, Atlanta, Georgia, June 13-14, 2013. c⃝2013 Association for Computational Linguistics Using the text to evaluate short answers for reading comprehension exercises Andrea Horbach, Alexis Palmer and Manfred Pinkal Department of Computational Linguistics, Saarland University, Saarbr¨ucken, Germany (andrea|apalmer|pinkal)@coli.uni-saarland.de Abstract Short answer questions for reading compre- hension are a common task in foreign lan- guage learning. Automatic short answer scor- ing is the task of automatically assessing the semantic content of a student’s answer, mark- ing it e.g. as correct or incorrect. While pre- vious approaches mainly focused on compar- ing a learner answer to some reference an- swer provided by the teacher, we explore the use of the underlying reading texts as addi- tional evidence for the classification. First, we conduct a corpus study targeting the links be- tween sentences in reading texts for learners of German and answers to reading comprehen- sion questions based on those texts. Second, we use the reading text directly for classifi- cation, considering three different models: an answer-based classifier extended with textual features, a simple text-based classifier, and a model that combines the two according to con- fidence of the text-based classification. The most promising approach is the first one, re- sults for which show that textual features im- prove classification accuracy. While the other two models do not improve classification ac- curacy, they do investigate the role of the text and suggest possibilities for developing auto- matic answer scoring systems with less super- vision needed from instructors. 1 Introduction Reading comprehension exercises are a common means of assessment for language teaching: students read a text in the language they are learning and are then asked to answer questions about the text. The types of questions asked of the learner may vary in their scope and in the type of answers they are de- signed to elicit; in this work we focus on “short an- swer” responses, which are generally in the range of 1–3 sentences. The nature of the reading comprehension task is that the student is asked to show that he or she has understood the text at hand. Questions focus on one or more pieces of information from the text, and cor- rect responses should contain the relevant semantic content. In the language learning context, responses classified as correct might still contain grammatical or spelling errors; the focus lies on the content rather than the form of the learner answer. Automatic scoring of short answer responses to reading comprehension questions is in essence a tex- tual entailment task, with the additional complica- tion that, in order to answer a question correctly, the learner must have identified the right portion of the text. It isn’t enough that a student answer is en- tailed by some part of the reading text; it must be entailed by the part of the text which is responsive to the question under discussion. Previous approaches to …

Content truncated to 500 words