AR-ASAG An ARabic Dataset for Automatic Short Answer Grading Evaluation

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 2634–2643 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 2634 AR-ASAG An ARabic Dataset for Automatic Short Answer Grading Evaluation Leila OUAHRANI, Djamal BENNOUAR Bouira University Algeria l_ouahrani@univ-blida.dz, djamal.bennouar@univ-bouira.dz Abstract Automatic short answer grading is a significant problem in E-assessment. Several models have been proposed to deal with it. Evaluation and comparison of such solutions need the availability of Datasets with manual examples. In this paper, we introduce AR-ASAG, an Arabic Dataset for automatic short answer grading. The Dataset contains 2133 pairs of (Model Answer, Student Answer) in several versions (txt, xml, Moodle xml and .db). We explore then an unsupervised corpus based approach for automatic grading adapted to the Arabic Language. We use COALS (Correlated Occurrence Analogue to Lexical Semantic) algorithm to create semantic space for word distribution. The summation vector model is combined to term weighting and common words to achieve similarity between a teacher model answer and a student answer. The approach is particularly suitable for languages with scarce resources such as Arabic language where robust specific resources are not yet available. A set of experiments were conducted to analyze the effect of domain specificity, semantic space dimension and stemming techniques on the effectiveness of the grading model. The proposed approach gives promising results for Arabic language. The reported results may serve as baseline for future research work evaluation. Keywords: Short Answer Grading, Dataset, Semantic Space, Semantic Similarity, Term Weighting, Arabic Language, Corpus, Stemming 1. Introduction Assessment is a key component of the teaching and learning process. The templates provided by most Computer-Aided Assessment Systems are multiple-choice questions, true/false questions and matching questions. Only basic support around the management of open-ended questions (short answer questions and essays) is offered by few others. Short answer questions(few words to a few sentences constructed in natural language) are more effective focusing on recall and reproduction (Anderson et al., 2001). However, their assessment is a complex and subjective process that requires the analysis and a deep understanding of a natural language text. Although Automatic Short Answer Grading systems (ASAG) have been studied for many years, their adoption in practice is not common due to their complexity (Liu et al., 2014). Unlike some systems which use information extraction with manually written patterns, templates, or machine learning to perform the ASAG task, we deal with the problem as a semantic similarity problem between the Student's Answer (SA) and the teacher's Model Answer (MA)(( Mohler and Mihalcea, 2009), (Mohler et al., 2011), (Gomaa and Fahmy, 2014b), (Gomaa and Fahmy, 2014a), (Zahran et al., 2015), (Magooda et al., 2016), (Bennouar, 2017)). There are two main approaches to determine the semantic similarity between two short texts: topological similarity and statistical similarity(corpus-based) (Mihalcea et al., 2006). Topological similarity uses data models containing information about concepts and their correlation (WordNet1, thesaurus, dictionaries). Statistical similarity uses vector state spaces to express correlations of words extracted from text corpora. Many solutions for …

Content truncated to 500 words