Evaluating Argumentative and Narrative Essays using Graphs

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1568–1578, Osaka, Japan, December 11-17 2016. Evaluating Argumentative and Narrative Essays using Graphs Swapna Somasundaran, Brian Riordan, Binod Gyawali and Su-Youn Yoon Educational Testing Service 660 Rosedale Road, Princeton NJ 08541, USA {ssomasundaran,briordan,bgyawali,SYoon}@ets.org Abstract This work investigates whether the development of ideas in writing can be captured by graph properties derived from the text. Focusing on student essays, we represent the essay as a graph, and encode a variety of graph properties including PageRank as features for modeling essay scores related to quality of development. We demonstrate that our approach improves on a state- of-the-art system on the task of holistic scoring of persuasive essays and on the task of scoring narrative essays along the development dimension. 1 Introduction Development, elaboration and exemplification are important writing skills that come into play in many different genres of writing. In a persuasive essay, to produce good arguments, it is important to substanti- ate a stance. Similarly, in a story, it is important to develop plot, character and events. Consequently, skill in development is evaluated in essay writing tasks at all levels of education, from primary school to the graduate level. For example, on the Graduate Record Examination (GRE), the scoring guidelines recom- mend that the top score be assigned to persuasive essays that “develop the position fully with compelling reasons and/or persuasive examples”1. Primary school assessments of narrative writing (i.e., stories and personal experiences, real or imagined) also test this skill. For instance, the scoring rubric for one of the U.S. Common Core State Standards tests2 has a dimension for “elaboration and development”, where top scores are given to essays where “experiences, characters, settings and/or events are clearly developed”. Previous research has investigated techniques for the automated assessment of essays by evaluating aspects of writing such as grammar, fluency, and coherence (Shermis and Burstein, 2013; Miltsakaki and Kukich, 2004; Attali and Burstein, 2006; Rus and Niraula, 2012; Stab and Gurevych, 2014b; Somasun- daran et al., 2014; Rahimi et al., 2015; Song et al., 2014; Farra et al., 2015). In this work, we investigate how to evaluate the development of ideas and exemplification. Specifically, we explore whether develop- ment is reflected in the structure of graphs constructed from the discourse proximity of essay concepts. We construct graphs for each essay where the essay’s concepts comprise the nodes and links are formed from concepts occurring in adjacent sentences. We then use properties of each graph, such as PageRank, to predict essay quality, including the quality of development. Our hypothesis is that this novel graph representation can help distinguish essays with well-developed ideas from essays lacking development and elaboration. We test the effectiveness of our approach on two different essay datasets: holistically scored persuasive essays and trait-scored narrative essays. Our results demonstrate that graph-based features are useful across genres for essay scoring, and improve the performance of models for both holistic scoring and trait scoring. With the addition of these features, …

Content truncated to 500 words