UKP-BIU: Similarity and Entailment Metrics for Student Response Analysis

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 285–289, Atlanta, Georgia, June 14-15, 2013. c⃝2013 Association for Computational Linguistics UKP-BIU: Similarity and Entailment Metrics for Student Response Analysis Torsten Zesch† Omer Levy§ Iryna Gurevych† Ido Dagan§ † Ubiquitous Knowledge Processing Lab § Natural Language Processing Lab Computer Science Department Computer Science Department Technische Universit¨at Darmstadt Bar-Ilan University Abstract Our system combines text similarity measures with a textual entailment system. In the main task, we focused on the influence of lexical- ized versus unlexicalized features, and how they affect performance on unseen questions and domains. We also participated in the pi- lot partial entailment task, where our system significantly outperforms a strong baseline. 1 Introduction The Joint Student Response Analysis and 8th Rec- ognizing Textual Entailment Challenge (Dzikovska et al., 2013) brings together two important dimen- sions of Natural Language Processing: real-world applications and semantic inference technologies. The challenge focuses on the domain of middle- school quizzes, and attempts to emulate the metic- ulous marking process that teachers do on a daily basis. Given a question, a reference answer, and a student’s answer, the task is to determine whether the student answered correctly. While this is not a new task in itself, the challenge focuses on em- ploying textual entailment technologies as the back- bone of this educational application. As a conse- quence, we formalize the question “Did the student answer correctly?” as “Can the reference answer be inferred from the student’s answer?”. This question can (hopefully) be answered by a textual entailment system (Dagan et al., 2009). The challenge contains two tasks: In the main task, the system must analyze each answer as a whole. There are three settings, where each one de- fines “correct” in a different resolution. The highest- resolution setting defines five different classes or “correctness values”: correct, partially correct, con- tradictory, irrelevant, non-domain. In the pilot task, critical elements of the answer need to be analyzed separately. Each such element is called a facet, and is defined as a pair of words that are critical in an- swering the question. As there is a substantial dif- ference between the two tasks, we designed sibling architectures for each task, and divide the main part of the paper accordingly. Our goal is to provide a robust architecture for stu- dent response analysis, that can generalize and per- form well in multiple domains. Moreover, we are interested in evaluating how well general-purpose technologies will perform in this setting. We there- fore approach the challenge by combining two such technologies: DKPro Similarity –an extensive suite of text similarity measures– that has been success- fully applied in other settings like the SemEval 2012 task on semantic textual similarity (B¨ar et al., 2012a) or reuse detection (B¨ar et al., 2012b). BIUTEE, the Bar-Ilan University Textual Entail- ment Engine (Stern and Dagan, 2011), which has shown state-of-the-art performance on recognizing textual entailment challenges. Our systems use both technologies to extract features, …

Content truncated to 500 words