A Statistical Model for Domain-Independent Text Segmentation

A Statistical Model for Domain-Independent Text Segmentation Masao Utiyama and Hitoshi Isahara Communications Research Laboratory 2-2-2 Hikaridai Seika-cho, Soraku-gun, Kyoto, 619-0289 Japan mutiyama@crl.go.jp and isahara@crl.go.jp Abstract We propose a statistical method that finds the maximum-probability seg- mentation of a given text. This method does not require training data because it estimates probabilities from the given text. Therefore, it can be applied to any text in any domain. An experi- ment showed that the method is more accurate than or at least as accurate as a state-of-the-art text segmentation sys- tem. 1 Introduction Documents usually include various topics. Identi- fying and isolating topics by dividing documents, which is called text segmentation, is important for many natural language processing tasks, in- cluding information retrieval (Hearst and Plaunt, 1993; Salton et al., 1996) and summarization (Kan et al., 1998; Nakao, 2000). In informa- tion retrieval, users are often interested in par- ticular topics (parts) of retrieved documents, in- stead of the documents themselves. To meet such needs, documents should be segmented into co- herent topics. Summarization is often used for a long document that includes multiple topics. A summary of such a document can be composed of summaries of the component topics. Identifi- cation of topics is the task of text segmentation. A lot of research has been done on text seg- mentation (Kozima, 1993; Hearst, 1994; Oku- mura and Honda, 1994; Salton et al., 1996; Yaari, 1997; Kan et al., 1998; Choi, 2000; Nakao, 2000). A major characteristic of the methods used in this research is that they do not require training data to segment given texts. Hearst (1994), for exam- ple, used only the similarity of word distributions in a given text to segment the text. Consequently, these methods can be applied to any text in any domain, even if training data do not exist. This property is important when text segmentation is applied to information retrieval or summarization, because both tasks deal with domain-independent documents. Another application of text segmentation is the segmentation of a continuous broadcast news story into individual stories (Allan et al., 1998). In this application, systems relying on supervised learning (Yamron et al., 1998; Beeferman et al., 1999) achieve good performance because there are plenty of training data in the domain. These systems, however, can not be applied to domains for which no training data exist. The text segmentation algorithm described in this paper is intended to be applied to the sum- marization of documents or speeches. Therefore, it should be able to handle domain-independent texts. The algorithm thus does not use any train- ing data. It requires only the given documents for segmentation. It can, however, incorporate train- ing data when they are available, as discussed in Section 5. The algorithm selects the optimum segmen- tation in terms of the probability defined by a statistical model. This is a new approach for domain-independent text segmentation. Previous approaches usually used lexical cohesion to seg- ment texts into topics. Kozima (1993), for exam- ple, used cohesion based …

Content truncated to 500 words