Comparing Multi-label Classification with Reinforcement Learning for Summarisation of Time-series Data

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 1231–1240, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Comparing Multi-label Classification with Reinforcement Learning for Summarisation of Time-series Data Dimitra Gkatzia, Helen Hastie, and Oliver Lemon School of Mathematical and Computer Sciences, Heriot-Watt University, Edinburgh {dg106, h.hastie, o.lemon}@hw.ac.uk Abstract We present a novel approach for automatic report generation from time-series data, in the context of student feedback genera- tion. Our proposed methodology treats content selection as a multi-label (ML) classification problem, which takes as in- put time-series data and outputs a set of templates, while capturing the dependen- cies between selected templates. We show that this method generates output closer to the feedback that lecturers actually gener- ated, achieving 3.5% higher accuracy and 15% higher F-score than multiple simple classifiers that keep a history of selected templates. Furthermore, we compare a ML classifier with a Reinforcement Learn- ing (RL) approach in simulation and using ratings from real student users. We show that the different methods have different benefits, with ML being more accurate for predicting what was seen in the training data, whereas RL is more exploratory and slightly preferred by the students. 1 Introduction Summarisation of time-series data refers to the task of automatically generating text from vari- ables whose values change over time. We con- sider the task of automatically generating feed- back summaries for students describing their per- formance during the lab of a Computer Science module over the semester. Students’ learning can be influenced by many variables, such as difficulty of the material (Person et al., 1995), other dead- lines (Craig et al., 2004), attendance in lectures (Ames, 1992), etc. These variables have two im- portant qualities. Firstly, they change over time, and secondly they can be dependent on or inde- pendent of each other. Therefore, when generating feedback, we need to take into account all vari- ables simultaneously in order to capture potential dependencies and provide more effective and use- ful feedback that is relevant to the students. In this work, we concentrate on content selec- tion which is the task of choosing what to say, i.e. what information is to be included in a report (Reiter and Dale, 2000). Content selection deci- sions based on trends in time-series data determine the selection of the useful and important variables, which we refer to here as factors, that should be conveyed in a summary. The decisions of factor selection can be influenced by other factors that their values are correlated with; can be based on the appearance or absence of other factors in the summary; and can be based on the factors’ be- haviour over time. Moreover, some factors may have to be discussed together in order to achieve some communicative goal, for instance, a teacher might want to refer to student’s marks as a moti- vation for increasing the number of hours studied. We frame content selection as a simple classifi- cation task: given a set of time-series …

Content truncated to 500 words