SemEval-2013 Task 7: The Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 263–274, Atlanta, Georgia, June 14-15, 2013. c⃝2013 Association for Computational Linguistics SemEval-2013 Task 7: The Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge Myroslava O. Dzikovska School of Informatics, University of Edinburgh Edinburgh, United Kingdom m.dzikovska@ed.ac.uk Rodney D. Nielsen University of North Texas Denton, TX, USA Rodney.Nielsen@UNT.edu Chris Brew Nuance Communications USA cbrew@acm.org Claudia Leacock CTB McGraw-Hill USA claudia leacock@mheducation.com Danilo Giampiccolo CELCT Italy giampiccolo@celct.it Luisa Bentivogli CELCT and FBK Italy bentivo@fbk.eu Peter Clark Vulcan Inc. USA peterc@vulcan.com Ido Dagan Bar-Ilan University Israel dagan@cs.biu.ac.il Hoa Trang Dang NIST hoa.dang@nist.gov Abstract We present the results of the Joint Student Response Analysis and 8th Recognizing Tex- tual Entailment Challenge, aiming to bring to- gether researchers in educational NLP tech- nology and textual entailment. The task of giving feedback on student answers requires semantic inference and therefore is related to recognizing textual entailment. Thus, we of- fered to the community a 5-way student re- sponse labeling task, as well as 3-way and 2- way RTE-style tasks on educational data. In addition, a partial entailment task was piloted. We present and compare results from 9 partic- ipating teams, and discuss future directions. 1 Introduction One of the tasks in educational NLP systems is pro- viding feedback to students in the context of exam questions, homework or intelligent tutoring. Much previous work has been devoted to the automated scoring of essays (Attali and Burstein, 2006; Sher- mis and Burstein, 2013), error detection and correc- tion (Leacock et al., 2010), and classification of texts by grade level (Petersen and Ostendorf, 2009; Shee- han et al., 2010; Nelson et al., 2012). In these appli- cations, NLP methods based on shallow features and supervised learning are often highly effective. How- ever, for the assessment of responses to short-answer questions (Leacock and Chodorow, 2003; Pulman and Sukkarieh, 2005; Nielsen et al., 2008a; Mohler et al., 2011) and in tutorial dialog systems (Graesser et al., 1999; Glass, 2000; Pon-Barry et al., 2004; Jor- dan et al., 2006; VanLehn et al., 2007; Dzikovska et al., 2010) deeper semantic processing is likely to be appropriate. Since the task of making and testing a full edu- cational dialog system is daunting, Dzikovska et al. (2012) identified a key subtask and proposed it as a new shared task for the NLP community. Student response analysis (henceforth SRA) is the task of labeling student answers with categories that could 263 Example 1 QUESTION You used several methods to separate and identify the substances in mock rocks. How did you separate the salt from the water? REF. ANS. The water was evaporated, leaving the salt. STUD. ANS. The water dried up and left the salt. Example 2 QUESTION Georgia found one brown mineral and one black mineral. How will she know which one is harder? REF. ANS. The harder mineral will leave a scratch on the less hard mineral. If the black mineral is …

Content truncated to 500 words