Towards Evaluating Narrative Quality In Student Writing

Towards Evaluating Narrative Quality In Student Writing Swapna Somasundaran1, Michael Flor1, Martin Chodorow2 Hillary Molloy3 Binod Gyawali1 Laura McCulla1 1Educational Testing Service, 660 Rosedale Road, Princeton, NJ 08541, USA 2Hunter College and the Graduate Center, CUNY, New York, NY 10065, USA 3Educational Testing Service, 90 New Montgomery Street, San Francisco, CA 94105, USA {ssomasundaran,mflor,hmolloy, bgyawali,LMcCulla}@ets.org martin.chodorow@hunter.cuny.edu Abstract This work lays the foundation for automated assessments of narrative quality in student writing. We first manually score essays for narrative-relevant traits and sub-traits, and measure inter-annotator agreement. We then explore linguistic features that are indicative of good narrative writing and use them to build an automated scoring system. Experiments show that our features are more effective in scoring specific aspects of narrative quality than a state-of-the-art feature set. 1 Introduction Narrative, which includes personal experiences and stories, real or imagined, is a medium of expression that is used from the very early stages of a child’s life. Narratives are also employed in various capac- ities in school instruction and assessment. For ex- ample, the Common Core State Standards, an ed- ucational initiative in the United States that details requirements for student knowledge in grades K- 12, employs literature/narratives as one of its three language arts genres. With the increased focus on automated evaluation of student writing in educa- tional settings (Adams, 2014), automated methods for evaluating narrative essays at scale are becoming increasingly important. Automated scoring of narrative essays is a chal- lenging area, and one that has not been explored ex- tensively in NLP research. Previous work on auto- mated essay scoring has focused on informational, argumentative, persuasive and source-based writing constructs (Stab and Gurevych, 2017; Nguyen and Litman, 2016; Farra et al., 2015; Somasundaran et al., 2014; Beigman Klebanov et al., 2014; Shermis and Burstein, 2013). Similarly, operational essay scoring engines (Attali and Burstein, 2006; Elliot, 2003) are geared towards evaluating language profi- ciency in general. In this work, we lay the ground- work and present the first results for automated scor- ing of narrative essays, focusing on narrative quality. One of the challenges in narrative quality anal- ysis is the scarcity of scored essays in this genre. We describe a detailed manual annotation study on scoring student essays along multiple dimensions of narrative quality, such as narrative development and narrative organization. Using a scoring rubric adapted from the U.S. Common Core State Stan- dards, we annotated 942 essays written for 18 differ- ent essay-prompts by students from three different grade levels. This data set provides a variety of story types and language proficiency levels. We measured inter-annotator agreement to understand reliability of scoring stories for traits (e.g., development) as well as sub-traits (e.g., plot development and the use of narrative techniques). A number of techniques for writing good stories are targeted by the scoring rubrics. We implemented a system for automatically scoring different traits of narratives, using linguistic features that capture some of those techniques. We investigated the effec- tiveness of each feature for scoring narrative traits and analyzed …

Content truncated to 500 words