A Semantic Feature-Wise Transformation Relation Network for Automatic Short Answer Grading

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6030–6040 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6030 A Semantic Feature-Wise Transformation Relation Network for Automatic Short Answer Grading Zhaohui Li and Yajur Tomar and Rebecca J. Passonneau Department of Computer Science and Engineering Pennsylvania State University {zjl5282, yst5012, rjp49}@psu.edu Abstract Automatic short answer grading (ASAG) is the task of assessing students’ short natural lan- guage responses to objective questions. It is a crucial component of new education plat- forms, and could support more wide-spread use of constructed response questions to re- place cognitively less challenging multiple choice questions. We propose a Semantic Feature-wise transformation Relation Network (SFRN) that exploits the multiple components of ASAG datasets more effectively. SFRN captures relational knowledge among the ques- tions (Q), reference answers or rubrics (R), and labeled student answers (A). A relation network learns vector representations for the elements of QRA triples, then combines the learned representations using learned seman- tic feature-wise transformations. We apply translation-based data augmentation to address the two problems of limited training data, and high data skew for multi-class ASAG tasks. Our model has up to 11% performance im- provement over state-of-the-art results on the benchmark SemEval-2013 datasets, and sur- passes custom approaches designed for a Kag- gle challenge, demonstrating its generality. 1 Introduction Educators at every level rely on classroom as- sessments to evaluate students’ knowledge, often through quizzes. Multiple choice questions can be graded automatically, but many studies have shown that short answer, constructed response questions provide greater benefit to students (Lee et al., 2011; McDaniel et al., 2007; Butler and Roediger, 2007; Clariana, 2003). Manual assessment of short as- nwer questions is time-consuming, and has bias and errors (Galhardi et al., 2020; Bejar, 2012). Au- tomatic Short Answer Grading (ASAG) applies NLP methods to reduce the assessment burden for short answer questions (Burrows et al., 2015), with potential to assist educators in providing more timely feedback to students on a preferred assess- ment method. With increased reliance on virtual learning environments and educational technology, the potential impact of ASAG has grown. The most common ASAG approach classifies students’ answers into two or more categories. The key challenge is that the classification problem is inherently relational, involving the relation of the student’s answer to the question, as well as to one or more reference answers. Another challenge is that existing datasets are relatively small, especially for the kinds of neural network models that per- form best on other NLP tasks. Much of the ASAG work is conducted in industry labs with proprietary methods and datasets (Wang et al., 2019; Liu et al., 2019). Creation of benchmark datasets, however, has fostered broader interest in the problem from the NLP community. The main benchmark dataset, SemEval-2013 (Dzikovska et al., 2013), covers multiple STEM domains, and has multiple classifi- cation tasks regarding both the number of classes, and the level of generalization required. For exam- ple, one task addresses unseen answers to known questions (UA), …

Content truncated to 500 words