Topic Segmentation for Short Texts

Topic Segmentation for Short Texts Tao-Hsing Chang Department of Computer and Information Science National Chiao Tung University 1001 Ta Hseuh Rd., Hsinchu, Taiwan 300, R.O.0 thchang@cis.nctu.edu.tw Chia-Honag Lee Department of Computer and Information Science National Chiao Tung University 1001 Ta Hseuh Rd., Hsinchu, Taiwan 300, R.O.0 chl@cis.nctu.edu.tw Abstract Topic segmentation, which aims to fmd the boundaries between topic blocks in a text, is an important task for semantic analysis of texts. Although different solutions have been proposed for the task, many limitations and difficulties exist in the approaches. In particular most of the methods do not work well for such case as short texts, internet news and student's writings. In this paper, we focus on the short texts and present a method for topic segmentation. It can overcome the limitations in previous works. In preliminary experiments, the method show the accuracy of topic segmentation is increased effectively. 1. Introduction Topic segmentation, which aims to fmd the boundaries between topic blocks in a text, is an important task for semantic analysis of texts. In general, they rely on such knowledge as word recurrence, collocations, thesaurus, linguistics cues, or the combination of those, to measure the similarity between sentences, and estimate whether topic shift occurs (Ferret(2002)). The studies are also classified into two schemes: one is based on locally comparing adjacent blocks of sentences, while the other method is based on considering topic block boundary decision as a global optimization (Bigi et.a1(1998)). Although the approaches for topic segmentation have been worked well for long texts, the assumptions it requires limit most of useful applications. First, before topic segmentation, the thresholds, coefficients, or parameters of formulas in some methods must be estimated beforehand depend on the characters of text sets or the experience of users. It not only limits applications to be fully automatic but also causes difficulty in many domains. These methods perform especially poor when they are applied to such short texts as internet news and student's writings (Ponte and Croft(1997)). Since keywords from sentences are quite few and not reliable for short texts, the errors in measure naturally occur more frequently. It results in rapid decrease of the accuracy of topic segmentation. The accuracy of these methods not only relies on the characteristics of the short texts, but also depends on the languages. For Chinese, the usage of punctuation marks such as comma mark is often ambiguous (Chen(1993)). For instance, the sentence ending with comma mark is not always considered a complete sentence on syntax or semantics. For incomplete sentences, the step of extracting reliable keywords is far more difficult. This causes, due to the characteristic of the language, further decrease of the accuracy of topic segmentation. Apparently, short texts are troublesome and quite difficult for existing methods in topic segmentation. However, given a theme, a huge collection of reference texts for short texts can easily be obtained. This large corpus or thesaurus allows us to overcome the above mentioned limitations and define similarity between sentences. This paper will present …

Content truncated to 500 words