A temporal model of text periodicities using Gaussian Processes

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 977–988, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics A temporal model of text periodicities using Gaussian Processes Daniel Preot¸iuc-Pietro, Trevor Cohn Department of Computer Science University of Sheffield Regent Court, 211 Portobello Street Sheffield, S1 4DP, United Kingdom {daniel,t.cohn}@dcs.shef.ac.uk Abstract Temporal variations of text are usually ig- nored in NLP applications. However, text use changes with time, which can affect many applications. In this paper we model peri- odic distributions of words over time. Focus- ing on hashtag frequency in Twitter, we first automatically identify the periodic patterns. We use this for regression in order to fore- cast the volume of a hashtag based on past data. We use Gaussian Processes, a state-of- the-art bayesian non-parametric model, with a novel periodic kernel. We demonstrate this in a text classification setting, assigning the tweet hashtag based on the rest of its text. This method shows significant improvements over competitive baselines. 1 Introduction Temporal changes in text corpora are central to our understanding of many linguistic and social phe- nomena. Social Media platforms and the digital- ization of libraries provides a vast body of times- tamped data. This allows studying of the complex temporal patterns exhibited by text usage includ- ing highly non-stationary distributions and period- icities. However, temporal effects have been mostly ignored by previous work on text analysis or at best dealt with by making strong assumptions such as smoothly varying parameters with time (Yogatama et al., 2011) or modelled using a simple uni-modal distri bution (Wang and McCallum, 2006). This pa- per develops a temporal model for classifying mi- croblog posts which explicitly incorporates mul- timodal periodic behaviours using Gaussian Pro- cesses (GPs). We expect text usage to follow multiple period- icities at different scales. For example, people on Social Media might talk about different topics dur- ing and after work on weekdays, talk every Friday about the weekend ahead, or comment about their favorite weekly TV show during its air time. Given this, text frequencies will display periodic patterns. This applies to other text related quantities like co- occurrence values or topic distributions over time, as well as applications outside NLP like user be- haviour (Preot¸iuc-Pietro and Cohn, 2013). Modelling temporal patterns and periodicities can be useful to tasks like text classification. For exam- ple a tweet containing ‘music’ is normally attributed to a general hashtag about music like #np (now play- ing). However, knowing time, if it occurs during the (weekly periodic) air time of ‘American Idol’ it is more likely for it to belong to #americanidol or if its mentioned in the days building up to the Video Music Awards to be assigned to #VMA. In NLP, temporal models have treated time in overly simplistic ways and without regard to period- icities. We propose a model that first broadly iden- tifies several types of temporal patterns: a) periodic, b) constant in time, c) falling out of use after enjoy- …

Content truncated to 500 words