An Improved Phrase-based Approach to Annotating and Summarizing Student Course Responses

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 53–63, Osaka, Japan, December 11-17 2016. An Improved Phrase-based Approach to Annotating and Summarizing Student Course Responses Wencan Luo† Fei Liu‡ Diane Litman† †University of Pittsburgh, Pittsburgh, PA 15260 ‡University of Central Florida, Orlando, FL 32816 {wencan, litman}@cs.pitt.edu feiliu@cs.ucf.edu Abstract Teaching large classes remains a great challenge, primarily because it is difficult to attend to all the student needs in a timely manner. Automatic text summarization systems can be leveraged to summarize the student feedback, submitted immediately after each lecture, but it is left to be discovered what makes a good summary for student responses. In this work we explore a new methodology that effectively extracts summary phrases from the student responses. Each phrase is tagged with the number of students who raise the issue. The phrases are evaluated along two di- mensions: with respect to text content, they should be informative and well-formed, measured by the ROUGE metric; additionally, they shall attend to the most pressing student needs, measured by a newly proposed metric. This work is enabled by a phrase-based annotation and highlight- ing scheme, which is new to the summarization task. The phrase-based framework allows us to summarize the student responses into a set of bullet points and present to the instructor promptly. 1 Introduction Effective teachers use student feedback to adjust their teaching strategies. Nowadays, in large classes, there is far too much feedback for a single teacher to manage and attend to. If different perspectives in the student feedback could be summarized and pressing issues identified, it would greatly enhance the teachers’ ability to make informed choices. In this work we seek to automatically summarize the student course feedback into a set of bullet points. Each bullet point corresponds to a phrase, tagged with the number of students who raise the issue. Our emphasis is on the textual feedback submitted by students after each lecture in response to two reflective prompts (Boud et al., 2013): 1) “Describe what you found most interesting in today’s class” and 2) “Describe what was confusing or needed more detail.” Education researchers have demonstrated that asking students to respond to reflection prompts can improve both teaching and learning (Van den Boom et al., 2004; Menekse et al., 2011). However, summarizing these responses for large classes (e.g., introductory STEM, MOOCs) remains costly, time-consuming, and an onerous task for humans (Mosteller, 1989). In our prior work, Luo and Litman (2015) (henceforth L&L) introduced the task of automatic summa- rization of student responses. The challenges of this task include 1) high lexical variety, because students tend to use different word expressions to communicate the same or similar meanings (e.g., “bike ele- ments” vs. “bicycle parts”), and 2) high length variety, as the student responses range from a single word to multiple sentences. To tackle the challenges, L&L proposed a phrase summarization framework con- sisting of three stages: phrase extraction, phrase clustering, and phrase ranking. The approach extracts noun phrases …

Content truncated to 500 words