EHU-ALM: Similarity-Feature Based Approach for Student Response Analysis

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 580–584, Atlanta, Georgia, June 14-15, 2013. c⃝2013 Association for Computational Linguistics EHU-ALM: Similarity-Feature Based Approach for Student Response Analysis Itziar Aldabe, Montse Maritxalar IXA NLP Group University of Basque Country (UPV-EHU) itziar.aldabe@ehu.es montse.maritxalar@ehu.es Oier Lopez de Lacalle University of Edinburgh IKERBASQUE, Basque Foundation for Science oier.lopezdelacalle@gmail.com Abstract We present a 5-way supervised system based on syntactic-semantic similarity features. The model deploys: Text overlap measures, WordNet-based lexical similarities, graph- based similarities, corpus-based similarities, syntactic structure overlap and predicate- argument overlap measures. These measures are applied to question, reference answer and student answer triplets. We take into account the negation in the syntactic and predicate- argument overlap measures. Our system uses the domain-specific data as one dataset to build a robust system. The results show that our system is above the median and mean on all the evaluation scenarios of the SemEval- 2013 task #7. 1 Introduction In this paper we describe our participation with a feature-based supervised system to the SemEval- 2013 task #7: The Joint Student Response Analy- sis and 8th Recognizing Textual Entailment Chal- lenge (Dzikovska et al., 2013). The goal of our participation is to build a generic system that is robust enough across domains and scenarios. domain-specific system requires new training ex-A amples when shifting to a new domain. However, domain-specific data is difficult to obtain and creat- ing new resources is expensive. We seek robustness by mixing the instances from BEETLE and SCIENTSBANK. We show our strategy is suitable to build a generic system that performs competitively on any domain in the 5-way task. The paper proceeds as follows. Section 2 de- scribes the system presenting the learning features and the runs. In Section 3 we show the optimiza- tion details, followed by the results (Section 4) and a preliminary error analysis (Section 5). 2 System description Our system aims for robustness using the domain- specific training data as one dataset. Therefore, we do not differentiate between examples from the given domains (BEETLE and SCIENTSBANK) when training the system. In contrast, our approach dintin- guishes between new questions (unseen answer vs. unseen question) as well as question types (how, what and why) by means of simple heuristics. The runs are organized according to different sys- tem designs. Although all the runs use the same fea- ture set, we split the training set to build more spe- cialized classifiers. Training examples are grouped depending on: i) the answer is unseen; ii) the ques- tion is unseen; and iii) the question type (i.e. what, how, why). Each run defines a framework to explore the different ways to approach the problem. While the first run is the simplest and is the most generic in nature, the third tries to split the task into simpler problems and creates more specialized classifiers. 2.1 Similarity learning features Our model is based on various text similarity fea- tures. Almost all of the measures …

Content truncated to 500 words