Chinese Novelty Mining

Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 1561–1570, Singapore, 6-7 August 2009. c⃝2009 ACL and AFNLP Chinese Novelty Mining Yi Zhang Nanyang Technological University 50 Nanyang Avenue Singapore 639798 yizhang@ntu.edu.sg Flora S. Tsai Nanyang Technological University 50 Nanyang Avenue Singapore 639798 fst1@columbia.edu Abstract Automated mining of novel documents or sentences from chronologically ordered documents or sentences is an open chal- lenge in text mining. In this paper, we describe the preprocessing techniques for detecting novel Chinese text and discuss the influence of different Part of Speech (POS) filtering rules on the detection per- formance. Experimental results on AP- WSJ and TREC 2004 Novelty Track data show that the Chinese novelty mining per- formance is quite different when choosing two dissimilar POS filtering rules. Thus, the selection of words to represent Chinese text is of vital importance to the success of the Chinese novelty mining. Moreover, we compare the Chinese novelty mining per- formance with that of English and investi- gate the impact of preprocessing steps on detecting novel Chinese text, which will be very helpful for developing a Chinese novelty mining system. 1 Introduction The bloom of information nowadays brings us rich useful information as well as tons of redundant in- formation in news articles, social networks (Tsai et al., 2009), and blogs (Chen et al., 2008). Novelty mining (NM), or novelty detection, aims at mining novel information from a chronologically ordered list of relevant documents/sentences. It can facil- itate users to quickly get useful information with- out going through a lot of redundant information, which is usually a tedious and time-consuming task. The process of detecting novel text contains three main steps, (i) preprocessing, (ii) cate- gorization, and (iii) novelty mining. The first step preprocesses the text documents/sentences by removing stop words, performing word stem- ming, implementing POS tagging etc. Categoriza- tion classifies each incoming document/sentence into its relevant topic bin. Then, within each topic bin containing a group of relevant docu- ments/sentences, novelty mining searches through the time sequence of documents/sentences and re- trieves only those with “novel” information. This paper focuses on applying document/sentence- level novelty mining on Chinese. In this task, we need to identify all novel Chinese text given groups of relevant documents/sentences. Novelty mining has been performed at three dif- ferent levels: event level, sentence level and doc- ument level (Li and Croft, 2005). Works on nov- elty mining at the event level originated from re- search on Topic Detection and Tracking (TDT), which is concerned with online new event detec- tion/first story detection (Allan et al., 1998; Yang et al., 2002; Stokes and Carthy, 2001; Franz et al., 2001; Brants et al., 2003). Research on doc- ument and sentence-level novelty mining aims to find relevant and novel documents/sentences given a stream of documents/sentences. Previous stud- ies on document and sentence-level novelty min- ing tend to apply some promising content-oriented techniques (Li and Croft, 2005; Allan et al., 1998; Yang et al., 1998; Zhang and Tsai, 2009). Simi- …

Content truncated to 500 words