Automatic learner summary assessment for reading comprehension

Proceedings of NAACL-HLT 2019, pages 2532–2542 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2532 Automatic learner summary assessment for reading comprehension Menglin Xia Amazon Research Cambridge∗ ximengli@amazon.co.uk Ekaterina Kochmar ALTA Institute Computer Laboratory University of Cambridge ek358@cl.cam.ac.uk Ted Briscoe ALTA Institute Computer Laboratory University of Cambridge ejb@cl.cam.ac.uk Abstract Automating the assessment of learner sum- maries provides a useful tool for assessing learner reading comprehension. We present a summarization task for evaluating non- native reading comprehension and propose three novel approaches to automatically assess the learner summaries. We evaluate our mod- els on two datasets we created and show that our models outperform traditional approaches that rely on exact word match on this task. Our best model produces quality assessments close to professional examiners. 1 Introduction Summarization is a well-established method of measuring reading proficiency in traditional En- glish as a second or other language (ESOL) assess- ment. It is considered an effective approach to test both cognitive and contextual dimensions of read- ing (Weir et al., 2013). However, due to the high time and cost demands of manual summary assess- ment, modern English exams usually replace the summarization task with multiple choice or short answer questions that are easier to score (Alder- son, 2005). Automating the assessment of learner summarization skills provides an efficient evalua- tion method for the quality of the learner summary and can lead to effective educational applications to enhance reading comprehension tasks. In this paper, we present a summarization task for evaluating non-native reading comprehension and propose three novel machine learning ap- proaches to assessing learner summaries. First, we extract features to measure the content simi- larity between the reading passage and the sum- mary. Secondly, we calculate a similarity matrix based on sentence-to-sentence similarity between ∗The work by the first author was done at the University of Cambridge prior to joining Amazon Research. the reading passage and the summary, and apply a Convolutional Neural Network (CNN) model to assess the summary quality using the similarity matrix. Thirdly, we build an end-to-end summa- rization assessment model using the Long Short Term Memory (LSTM) model. Finally, we com- bine the three approaches in a single system using a simple parallel ensemble modeling technique. We compiled two datasets to evaluate our models, and we release this data with the paper. We show that our models outperform traditional approaches that rely on exact word match on the task and that our best model produces quality assessments close to professional examiners. 2 Related Work Most of the previous studies on summary assess- ment are aimed at evaluating automated summa- rization systems (Lin, 2004; Lin and Hovy, 2003; Nenkova et al., 2007). In contrast to this line of work, our goal is to assess human-written sum- maries rather than machine-generated ones. Within the educational domain, several applica- tions have been developed to help students with their writing summarization skills. Summary Street (Wade-Stein and Kintsch, 2004) is an edu- cational software designed for children to develop …

Content truncated to 500 words