Predicting Desirable Revisions of Evidence and Reasoning in Argumentative Writing

Findings of the Association for Computational Linguistics: EACL 2023, pages 2550–2561 May 2-6, 2023 ©2023 Association for Computational Linguistics Predicting Desirable Revisions of Evidence and Reasoning in Argumentative Writing Tazin Afrin and Diane Litman University of Pittsburgh Pittsburgh, Pennsylvania 15260 {taa74,dlitman}@pitt.edu Abstract We develop models to classify desirable evi- dence and desirable reasoning revisions in stu- dent argumentative writing. We explore two ways to improve classifier performance – using the essay context of the revision, and using the feedback students received before the revision. We perform both intrinsic and extrinsic evalua- tion for each of our models and report a qual- itative analysis. Our results show that while a model using feedback information improves over a baseline model, models utilizing context - either alone or with feedback - are the most successful in identifying desirable revisions. 1 Introduction Successful essay writing by students typically in- volves multiple rounds of revision and assistance from teachers, peers, or automated writing evalua- tion (AWE) systems. Natural language processing (NLP) has become a key component of AWE sys- tems, with NLP being used to assess the content and structure of student writing and to automat- ically provide formative feedback (Beigman Kle- banov and Madnani, 2020; Zhang et al., 2016; Writ- ing Mentor, 2016; Wang et al., 2020). While some students produce revised texts that are in line with the feedback automatically generated by a system or provided by other humans, other students either ignore the feedback or are unsuccessful in their feedback implementation attempts (Wang et al., 2020). Hence, analyzing student revisions in terms of their desirability for improving essay quality is important. The development of AWE systems that leverage NLP to analyze a revision’s alignment to feedback messages is one approach to convey to students a sense of a good revision direction. Our research focuses on the automatic classi- fication of desirable and undesirable revisions of evidence use and reasoning 1 in argumentative writ- 1Such revisions of text content are generally considered most important in revising (Faigley and Witte, 1981). ing. Argumentative writing is a skill that students need to develop to be strong writers and learners. By evidence use, we refer to examples and details that students use to support an argument. By rea- soning, we refer to how evidence is explained and linked to an overall argument. Desirable revisions (e.g., add relevant evidence) are student revisions that have hypothesized utility in improving an essay in response to feedback (e.g., add more evidence), while undesirable revisions (e.g., add irrelevant ev- idence) do not have such hypothesized utility. Table 1 shows example desirable and undesirable revisions of evidence and reasoning from original to revised drafts of an essay aligned at the sentence- level. In response to the feedback shown at the top of Table 1, the student adds both reasoning and evidence. Sentences 3, 5, and 9 are added desir- able reasoning, desirable evidence, and undesirable reasoning respectively. The student also modified fluency in other sentences which is not shown here. Sentences 1, …

Content truncated to 500 words