Clustering Microtext Streams for Event Identification

International Joint Conference on Natural Language Processing, pages 719–725, Nagoya, Japan, 14-18 October 2013. Clustering Microtext Streams for Event Identification Jie Yin Computational Informatics CSIRO, Australia Jie.Yin@csiro.au Abstract The popularity of microblogging systems has resulted in a new form of Web data – microtext – which is very different from conventional well-written text. Microtext often has the characteristics of informal- ity, brevity, and varied grammar, which poses new challenges in applying tradi- tional clustering algorithms to analyze mi- crotext. In this paper, we propose a novel two-phase approach for clustering stream- ing microtext, in particular Twitter mes- sages, into event-based clusters. In the on- line phase, an incremental process is ap- plied to discover base clusters and main- tain detailed summary statistics. Upon de- mand for any user-specified time horizons, an offline phase is triggered to merge re- lated clusters together. We demonstrate that our proposed approach can achieve better clustering accuracy than state-of- the-art methods. Introduction Microtext is a newly emerging type of Web data which is generated in enormous volumes with the proliferation of online microblogging systems. These systems, such as Twitter and Facebook, pro- vide a light-weight, easy form of communication that enables individuals around the globe to share information and express their opinions in fluid and less formal ways. Microtext streams gener- ated from these sites offer a rich source of real- time information about a wide variety of real- world events, ranging from planned occurrences such as political campaigns or sports games, to unexpected incidents such as earthquakes or ter- rorist riots. To provide insight into user-generated content broadcast in microtext streams, clustering approaches have demonstrated great potential for identifying what topics people are talking about and tracking how events unfold over time. Clustering microtext streams poses a number of new challenges, due to short, noisy and informal nature of microtext [Ellen, 2011]. First, clustering techniques should be scalable to the sheer volume of data generated in microblogging systems. Twit- ter, for example, generates over 400 million tweets per day in early 2013. Thus, it is crucial to develop efficient clustering algorithms that can handle such massive amounts of streaming data. Second, mi- crotext often has the characteristic of informal- ity, brevity, varied grammar, and free-style. De- pending on various personal style or background knowledge, people tend to use different words to convey the same or similar meanings, when writ- ing about a particular event. Therefore, it is highly desirable to design effective clustering algorithms that can discover event-based clusters over time. To cope with the sparsity and brevity of micro- text, different methods have been proposed for mi- crotext clustering in recent years. The majority of previous work has primarily focused on cluster- ing a static collection of short documents [Rangrej et al., 2011, Tsur et al., 2012], or on using surface features to compute pairwise similarity between microtext [Reuter et al., 2011, Li et al., 2012]. However, the challenge of how to effectively clus- ter microtext in dynamic data streams has not been …

Content truncated to 500 words