Automated Scoring: Beyond Natural Language Processing

Proceedings of the 27th International Conference on Computational Linguistics, pages 1099–1109 Santa Fe, New Mexico, USA, August 20-26, 2018. 1099 Automated Scoring: Beyond Natural Language Processing Nitin Madnani Aoife Cahill Educational Testing Service Princeton, NJ, 08541 USA {nmadnani,acahill}@ets.org Abstract In this position paper, we argue that building operational automated scoring systems is a task that has disciplinary complexity above and beyond standard competitive shared tasks which usu- ally involve applying the latest machine learning techniques to publicly available data in order to obtain the best accuracy. Automated scoring systems warrant significant cross-discipline collabo- ration of which natural language processing and machine learning are just two of many important components. Such systems have multiple stakeholders with different but valid perspectives that can often times be at odds with each other. Our position is that it is essential for us as NLP re- searchers to understand and incorporate these perspectives into our research and work towards a mutually satisfactory solution in order to build automated scoring systems that are accurate, fair, unbiased, and useful. 1 What is Automated Scoring? Automated scoring is an NLP application usually deployed in the educational domain. It involves auto- matically analyzing a student’s response to a question and generating either (a) a score in order to assess the student’s knowledge and/or other skills and/or (b) some actionable feedback on how the student can improve the response (Page, 1966; Burstein et al., 1998; Burstein et al., 2004; Zechner et al., 2009; Bern- stein et al., 2010). It is considered an NLP application since typically the core technology behind the automated analysis of the student response enlists NLP techniques. The student responses can include essays, short answers, or spoken responses and the two most common kinds of automated scoring are the automated evaluation of writing quality and content knowledge. Both the scores and feedback are usually based on linguistic characteristics of the responses including but not limited to: (i) Lower-level errors in the response (e.g., pronunciation errors in spoken responses and grammati- cal/spelling errors in written responses), (ii) The discourse structure and/or organization of the response, (iii) Relevance of the response to the question that was asked. 2 Motivation Over the last few years, there has been a significant increase in the number of NLP conference and workshop publications on the task of automated scoring of student responses (Burrows et al., 2015; Zesch et al., 2015; Sultan et al., 2016). Much of this increase in interest stems from the public availability of fairly large datasets containing scored human responses as part of shared tasks and public contests (the ASAP1 and ASAP22 Kaggle shared tasks, the Powergrading dataset (Basu et al., 2013), and the SemEval This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/ 1https://www.kaggle.com/c/asap-aes 2https://www.kaggle.com/c/asap-sas 1100 2013 Shared Task (Dzikovska et al., 2013)). Although this increase in interest is well-motivated and has led to useful technical advances, it has also propagated the impression that the task of automated scoring can be a …

Content truncated to 500 words