Unsupervised Cross-Lingual Transfer of Structured Predictors without Source Data

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 191 - 205 July 10-15, 2022 ©2022 Association for Computational Linguistics Do Deep Neural Nets Display Human-like Attention in Short Answer Scoring? Zijie Zeng, Xinyu Li, Dragan Gaševi´c and Guanliang Chen∗ Centre for Learning Analytics, Monash University Melbourne, Victoria, Australia {Zijie.Zeng, Xinyu.Li, Dragan.Gasevic, Guanliang.Chen}@monash.edu Abstract Deep Learning (DL) techniques have been in- creasingly adopted for Automatic Text Scoring in education. However, these techniques often suffer from their inabilities to explain and jus- tify how a prediction is made, which, unavoid- ably, decreases their trustworthiness and hin- ders educators from embracing them in prac- tice. This study aimed to investigate whether (and to what extent) DL-based graders align with human graders regarding the important words they identify when marking short an- swer questions. To this end, we first conducted a user study to ask human graders to manually annotate important words in assessing answer quality and then measured the overlap between these human-annotated words and those identi- fied by DL-based graders (i.e., those receiving large attention weights). Furthermore, we ran a randomized controlled experiment to explore the impact of highlighting important words de- tected by DL-based graders on human grading. The results showed that: (i) DL-based graders, to a certain degree, displayed alignment with human graders no matter whether DL-based graders and human graders agreed on the qual- ity of an answer; and (ii) it is possible to fa- cilitate human grading by highlighting those DL-detected important words, though further investigations are necessary to understand how human graders exploit such highlighted words. 1 Introduction Automatic Text Scoring refers to the task of apply- ing computational techniques to score written text based on certain grading criteria (Alikaniotis et al., 2016). Since its inception (Page, 1966), Automatic Text Scoring has been actively investigated and ap- plied to assist educators in scoring student-written text, e.g., short answer questions and essays, which are often referred to as Automated Short Answer Scoring (ASAS) (Brew and Leacock, 2013) and Es- say Scoring (Rodriguez et al., 2019). Driven by the ∗ Corresponding author. great success of Deep Learning (DL) techniques in various NLP tasks, researchers have endeavored to apply them to construct ASAS systems in re- cent years (Xia et al., 2020; Sung et al., 2019b,a), some of which displayed performance compara- ble to human graders. For instance, (Xia et al., 2020) showed that the average performance of an attention-based bidirectional LSTM model could be up to 0.71 (measured by the metric Quadratic Weighted Kappa) in the ASAS competition orga- nized by the Hewlett Foundation, which can be deemed as achieving a substantial agreement with human graders. Though being effective, DL-based ASAS sys- tems have been widely plagued by the inability to explain how the quality of an answer is graded. The lack of understanding the underlying work- ing mechanism of these systems, beyond question, may stop educators from adopting them in teach- ing practice as there remain …

Content truncated to 500 words