Thread Cleaning and Merging for Microblog Topic Detection

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 589–597, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP Thread Cleaning and Merging for Microblog Topic Detection Jianfeng Zhang1,2, Yunqing Xia1, Bin Ma1,2, Jianmin Yao2, and Yu Hong2 1Dept. of Comp. Sci. & Tech., Tsinghua University, Beijing 100084, China yqxia@tsinghua.edu.cn 2School of Comp. Sci. & Tech., Soochow University, Suzhou 215104, China {jfzhang, bma, jyao, hongy}@suda.edu.cn Abstract As a classic natural language processing technology, topic detection recently attracts more research interests due largely to the rapid development of microblog. The most challenging issue in microblog topic detection is sparse data problem. In this paper, the temporal-author-topic (TAT) model is designed to accomplish microblog topic detection in two phases. In the first phase, the TAT model is applied to clean the thread, namely, to filter noisy microblog texts out of each thread. In the second phase, microblog texts within each thread are merged to form the thread text so that the TAT model is applied to find global topics. The new approach differs from the Hierarchical Agglomerative Clustering (HAC) algorithm by making use of microblog threads to overcome the sparse data problem. Experimental results justify our claims. 1 Introduction Topic detection is the technique that discovers the latent topics from the given collection of text. Originated from the famous TDT evaluation workshop1, topic detection research has attracted intensive and persistent interests from governments with security purpose. With the rapid advance of the Internet, the Web content becomes so plentiful that people start to explore how to make good use of the content with commercial purpose. Very recently, microblog becomes surprisingly popular. According to the recent Twitter statistics, 155 million tweets are created per day on average2. This leads to huge research passion on the microblog content. Theoretically, topic detection from microblog text is similar to that from news articles. However, the microblog text is rather different 1 http://projects.ldc.upenn.edu/TDT/ 2 http://techcrunch.com/2011/04/06/twitter-q1-stats/ from news articles. According to Ellen et al. (2011), microblog text is a typical microtext. Compared to regular long text such as news article, the microblog text exhibits the following characteristics. � Short. Every microblog text finishes in less than 140. In fact, most texts are only a sentence or even a phrase. � Informal. Spoken language is typically used, usually containing abbreviations and misspellings. � Semi-structured. Each microblog text contains text as well as author and time. � Highly contextual. Most microblog texts are created by replying or evaluating the existing texts. Meanwhile, they are replied or evaluated by others. � Conversational. The microblog texts are usually naturally organized by thousands of conversation threads. In the thread, we name the top text by head posting, and the remaining texts by followup postings. The following challenges in microblog text processing are worth noting. Firstly, microblog texts, especially the followup postings, contain very few characters. This inevitably leads to serious sparse data problem when machine learning algorithms are adopted to handle the microblog texts. Secondly, grammar is usually informal …

Content truncated to 500 words