SOFTCARDINALITY: Hierarchical Text Overlap for Student Response Analysis

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 280–284, Atlanta, Georgia, June 14-15, 2013. c⃝2013 Association for Computational Linguistics SOFTCARDINALITY: Hierarchical Text Overlap for Student Response Analysis Sergio Jimenez, Claudia Becerra Universidad Nacional de Colombia Ciudad Universitaria, edificio 453, oficina 114 Bogotá, Colombia sgjimenezv@unal.edu.co cjbecerrac@unal.edu.co Alexander Gelbukh CIC-IPN Av. Juan Dios Bátiz, Av. Mendizábal, Col. Nueva Industrial Vallejo CP 07738, DF, México gelbukh@gelbukh.com Abstract In this paper we describe our system used to participate in the Student-Response-Analysis task-7 at SemEval 2013. This system is based on text overlap through the soft cardinality and a new mechanism for weight propagation. Al- though there are several official performance measures, taking into account the overall ac- curacy throughout the two availabe data sets (Beetle and SciEntsBank), our system ranked first in the 2 way classification task and sec- ond in the others. Furthermore, our sys- tem performs particularly well with “unseen- domains” instances, which was the more chal- lenging test set. This paper also describes an- other system that integrates this method with the lexical-overlap baseline provided by the task organizers obtaining better results than the best official results. We concluded that the soft cardinality method is a very competitive baseline for the automatic evaluation of stu- dent responses. 1 Introduction The Student-Response-Analysis (SRA) task consist in provide assessments of the correctness of student answers (A), considering their corresponding ques- tions (Q) and reference answers (RA) (Dzikovska et al., 2012). SRA is the task-7 in the SemEval 2013 evaluation campaign (Dzikovska et al., 2013). The method used in our participation was basically text overlap based on the soft cardinality (Jimenez et al., 2010) plus a machine learning classifier. This method did not use any information external to the data sets except for a stemmer and a list of stop words. The soft cardinality is a general model for object comparison that has been tested at text applications. Particularly, this text overlap approach has provided strong baselines for several applications, i.e. entity resolution (Jimenez et al., 2010), semantic textual similarity (Jimenez et al., 2012a), cross-lingual tex- tual entailment (Jimenez et al., 2012b), information retrieval, textual entailment and paraphrase detec- tion (Jimenez and Gelbukh, 2012). A brief descrip- tion of the soft cardinality is presented in the next section. The data for SRA consist of two data sets Bee- tle (5,199 instances) and SciEntsBank (10,804 in- stances) divided into training and test sets (76%- 24% for Beetle and 46%-54% SciEntsBank). In ad- dition, the test part of Beetle data set was divided into two test sets: “unseen answers” (35%) and “un- seen questions” (65%). Similarity, SciEntsBank test part is divided into “unseen answers” (9%), “unseen questions” (13%) and “unseen domains” (78%). All texts are in English. The challenge consists in predicting for each in- stance triple (Q, A, RA) an assessment of correct- ness for the student’s answer. Three levels of detail are considered for this assessment: 2 way (correct and incorrect), 3 …

Content truncated to 500 words