Improving Text-based Early Prediction by Distillation from Privileged Time-Series Text

Improving Text-based Early Prediction by Distillation from Privileged Time-Series Text Jinghui Liua,b Daniel Capurroa Anthony Nguyenb Karin Verspoorc,a a School of Computing and Information Systems, The University of Melbourne b Australian e-Health Research Centre, CSIRO c School of Computing Technologies, RMIT jinghui.liu@student.unimelb.edu.au dcapurro@unimelb.edu.au, anthony.nguyen@csiro.au karin.verspoor@rmit.edu.au Abstract Modeling text-based time-series to make pre- diction about a future event or outcome is an important task with a wide range of applica- tions. The standard approach is to train and test the model using the same input window, but this approach neglects the data collected in longer input windows between the prediction time and the final outcome, which are often available during training. In this study, we propose to treat this neglected text as privileged informa- tion available during training to enhance early prediction modeling through knowledge distil- lation, presented as Learning using Privileged tIme-sEries Text (LuPIET). We evaluate the method on clinical and social media text, with four clinical prediction tasks based on clin- ical notes and two mental health prediction tasks based on social media posts. Our re- sults show LuPIET is effective in enhancing text-based early predictions, though one may need to consider choosing the appropriate text representation and windows for privileged text to achieve optimal performance. Compared to two other methods using transfer learning and mixed training, LuPIET offers more sta- ble improvements over the baseline, standard training. As far as we are concerned, this is the first study to examine learning using privileged information for time-series in the NLP context. 1 Introduction Time-series forecasting, or early prediction, is an important machine learning task with a wide range of applications, such as weather predic- tion (Krasnopolsky and Fox-Rabinovitz, 2006; Es- peholt et al., 2022) and stock forecasting (Xu and Cohen, 2018; Sharma et al., 2017). Predicting fu- ture events or outcomes would enable timely re- sponses that can bring significant social and eco- nomic benefits. Meanwhile, most existing works on forecasting or early prediction use structured measurements or features as input (Steyerberg, 2009), and studies to leverage unstructured text Training Testing Training Testing 1day Model 1 3day 7day 1day Same train/test window Model 2 Model 3 3day 7day (a) Standard training Knowledge Distillation 1day 3day Model 1day Teacher Model 7day Training Testing Or (b) LuPIET 1day 3day Model Model 1day Model Transfer 7day Training Testing Or (c) Transfer learning 1day Model 3day 7day 1day Training Testing Mixed Samples (d) Mixed training Figure 1: Different methods to leverage later data from the time-series to assist early prediction. LuPIET refers to learning using privileged time-series text in training via knowledge distillation. Here 1-day is the baseline prediction window at test time. Models may leverage data from the prolonged training windows, e.g., 3-day, to enhance the performance for the shorter test window. to explore temporal patterns are still scarce (Assale et al., 2019). Moreover, user-generated, domain- specific textual data, such as clinical and social me- dia texts, can be noisy and complex to model (Bald- win et al., 2013; Huang et al., …

Content truncated to 500 words