Did I really mean that? Applying automatic summarisation techniques to formative feedback

Proceedings of Recent Advances in Natural Language Processing, pages 277–284, Hissar, Bulgaria, 7-13 September 2013. Did I really mean that? Applying automatic summarisation techniques to formative feedback Debora Field, Stephen Pulman University of Oxford, Oxford, UK debora, stephen.pulman@cs.ox.ac.uk Nicolas Van Labeke, Denise Whitelock, John T.E. Richardson The Open University, Milton Keynes, UK Nicolas.Vanlabeke, Denise.Whitelock, John.T.E.Richardson@open.ac.uk Abstract (2)This paper reports on an application that delivers automated formative feed- back designed to help university students improve their assignments. (3)The aim of the system is to improve the confidence and skills of the user by promoting self- directed learning through metacognition. (4)The system focuses on the content of an essay by using automatic summarisation techniques, automatic structure recogni- tion, diagrams, animations, and interactive exercises that promote reflection. (15)The system is currently undergoing initial ex- ploratory rounds of testing by ex-student volunteers and will be the subject of two full-scale empirical evaluations starting in September 2013. (1)The main claims of this paper are the application and adap- tation of graph-based key word and key sentence ranking methods for a novel pur- pose, and ensuing observations concerning the suitability of two different centrality algorithms for the purposes of key word extraction. 1 Introduction A fundamental problem in distance education is student attrition, particularly during the early months of enrolment, which appears to be largely due to low morale. Graduation rates at distance- learning institutions are often less than 20% (Simpson, 2012). Poor retention is evident at the level of individual modules or course units, where completion rates may be as low as 60–70%, or even lower for particular groups of students, such as those from ethnic minorities (Richardson, 2012). Some students who have dropped out of Open University courses have reported that the reason they left was a conviction of their own in- adequacy when faced with completing course as- signments. These reports are backed up by the drop-out rate that occurs just before the first as- signment is due, which, for some courses, is typi- cally as high as 30%. It appears, then, that there is a need for strate- gies that increase students’ confidence and skills during the early weeks of enrolment. The ideal strategy would be to provide frequent consulta- tions with human tutors, but resource implications dictate that this is not a viable solution. (10)We therefore decided to build an automated formative feedback system that could provide students with immediate feedback on the quality of their draft assignment essays and reports. (11)The purpose and design of our system are very different from existing automated assess- ment systems. (6)The system is primarily focused on user understanding and self-directed learning, rather than on essay improvement, and it engages the user on matters of content, rather than pointing out failings in grammar, style, and structure. (18)An early prototype of the system (called ‘openEssayist’) is implemented, and is currently undergoing first rounds of user testing. (17)Results from the user testing will inform improvements to the system, which is to be used this September by real …

Content truncated to 500 words