Automated Topical Component Extraction Using Neural Network Attention Scores from Source-based Essay Scoring

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8569–8584 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8569 Automated Topical Component Extraction Using Neural Network Attention Scores from Source-based Essay Scoring Haoran Zhang Department of Computer Science University of Pittsburgh Pittsburgh, PA 15260 colinzhang@cs.pitt.edu Diane Litman Department of Computer Science & LRDC University of Pittsburgh Pittsburgh, PA 15260 litman@cs.pitt.edu Abstract While automated essay scoring (AES) can re- liably grade essays at scale, automated writing evaluation (AWE) additionally provides forma- tive feedback to guide essay revision. How- ever, a neural AES typically does not provide useful feature representations for supporting AWE. This paper presents a method for link- ing AWE and neural AES, by extracting Top- ical Components (TCs) representing evidence from a source text using the intermediate out- put of attention layers. We evaluate perfor- mance using a feature-based AES requiring TCs. Results show that performance is compa- rable whether using automatically or manually constructed TCs for 1) representing essays as rubric-based features, 2) grading essays. 1 Introduction Automated essay scoring (AES) systems reliably grade essays at scale, while automated writing eval- uation (AWE) systems additionally provide forma- tive feedback to guide revision. Although neural networks currently generate state-of-the-art AES results (Alikaniotis et al., 2016; Taghipour and Ng, 2016; Dong et al., 2017; Farag et al., 2018; Jin et al., 2018; Li et al., 2018; Tay et al., 2018; Zhang and Litman, 2018), non-neural AES create feature rep- resentations more easily useable by AWE (Roscoe et al., 2014; Foltz and Rosenstein, 2015; Crossley and McNamara, 2016; Woods et al., 2017; Madnani et al., 2018; Zhang et al., 2019). We believe that neural AES can also provide useful information for creating feature representations, e.g., by exploiting information in the intermediate layers. Our work focuses on a particular source-based essay writing task called the response-to-text as- sessment (RTA) (Correnti et al., 2013). Recently, an RTA AWE system (Zhang et al., 2019) was built by extracting rubric-based features related to the use of Topical Components (TCs) in an essay. How- ever, manual expert effort was first required to cre- ate the TCs. For each source, the TCs consist of a comprehensive list of topics related to evidence which include: 1) important words indicating the set of evidence topics in the source, and 2) phrases representing specific examples for each topic that students need to find and use in their essays. To eliminate this expert effort, we propose a method for using the interpretable output of the attention layers of a neural AES for source-based essay writing, with the goal of extracting TCs. We evaluate this method by using the extracted TCs to support feature-based AES for two RTA source texts. Our results show that 1) the feature-based AES with TCs manually created by humans is matched by our neural method for generating TCs , and 2) the values of the rubric-based essay features based on automatic TCs are highly correlated with human Evidence scores. 2 Related …

Content truncated to 500 words