Give Me More Feedback II: Annotating Thesis Strength and Related Attributes in Student Essays

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3994–4004 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3994 Give Me More Feedback II: Annotating Thesis Strength and Related Attributes in Student Essays Zixuan Ke Hrishikesh Inamdar Hui Lin Vincent Ng Human Language Technology Research Institute University of Texas at Dallas Richardson, TX 75083-0688 {zixuan,hui,vince}@hlt.utdallas.edu, hai160030@utdallas.edu Abstract While the vast majority of existing work on automated essay scoring has focused on holis- tic scoring, researchers have recently begun work on scoring specific dimensions of essay quality. Nevertheless, progress in dimension- specific essay scoring research is hindered in part by the lack of annotated corpora. To facil- itate advances in this area of research, we de- sign a rubric for scoring an important, yet un- explored dimension of persuasive essay qual- ity, thesis strength, and annotate a corpus of es- says with thesis strength scores. We addition- ally identify the attributes that could impact thesis strength and annotate the essays with the values of these attributes, which, when pre- dicted by computational models, could pro- vide feedback to students on why her essay re- ceives a particular thesis strength score. 1 Introduction Recent work on automated essay scoring has largely focused on holistic scoring, which sum- marizes the quality of an essay with a single score (e.g., Taghipour and Ng (2016), Dong et al. (2017), Wang et al. (2018)). There are at least two reasons for this focus. First, corpora manu- ally annotated with holistic scores such as the one used in the Kaggle-sponsored ASAP competition1 are publicly available, facilitating the training and evaluation of holistic essay scoring engines. Sec- ond, holistic scoring technologies are commer- cially valuable: being able to successfully auto- mate the scoring of the millions of essays written for aptitude tests such as SAT, GRE, and GMAT every year can save a lot of manual grading effort. However, holistic essay scoring technologies are far from adequate for use in classroom settings, where providing students with feedback on how to improve their essays is of utmost importance. 1https://www.kaggle.com/c/asap-aes Specifically, merely returning a low holistic score to an essay provides essentially no feedback to its author on which aspect(s) of the essay contributed to the low score and how it can be improved. Re- cently, researchers have attempted to score a par- ticular dimension of essay quality such as coher- ence (Miltsakaki and Kukich, 2004), technical er- rors, relevance to prompt (Higgins et al., 2004; Louis and Higgins, 2010; Persing and Ng, 2014), organization (Persing et al., 2010), thesis clarity (Persing and Ng, 2013), and argument persuasive- ness (Persing and Ng, 2015; Ke et al., 2018). Au- tomated systems that provide instructional feed- back along multiple dimensions of essay quality such as Criterion (Burstein et al., 2004) have also begun to emerge. Providing scores along different dimensions of essay quality could help an author identify which aspects of her essay need improve- ments. Unfortunately, progress in dimension- specific essay scoring …

Content truncated to 500 words