An Automatic Method of Finding Topic Boundaries

AN AUTOMATIC METHOD OF FINDING TOPIC BOUNDARIES Jeffrey C. Reynar* Department of Computer and Information Science University of Pennsylvania Philadelphia, Pennsylvania, USA j creynar@unagi.cis.upenn.edu Abstract This article outlines a new method of locating discourse boundaries based on lexical cohesion and a graphical technique called dotplotting. The application of dot- plotting to discourse segmentation can be performed ei- ther manually, by examining a graph, or automatically, using an optimization algorithm. The results of two ex- periments involving automatically locating boundaries between a series of concatenated documents are pre- sented. Areas of application and future directions for this work are also outlined. Introduction In general, texts are "about" some topic. That is, the sentences which compose a document contribute infor- mation related to the topic in a coherent fashion. In all but the shortest texts, the topic will be expounded upon through the discussion of multiple subtopics. Whether the organization of the text is hierarchical in nature, as described in (Grosz and Sidner, 1986), or linear, as examined in (Skorochod'ko, 1972), boundaries between subtopics will generally exist. In some cases, these boundaries will be explicit and will correspond to paragraphs, or in longer texts, sec- tions or chapters. They can also be implicit. Newspa- per articles often contain paragraph demarcations, but less frequently contain section markings, even though lengthy articles often address the main topic by dis- cussing subtopics in separate paragraphs or regions of the article. Topic boundaries are useful for several different tasks. Hearst and Plaunt (1993) demonstrated their usefulness for information retrieval by showing that segmenting documents and indexing the resulting subdocuments improves accuracy on an information retrieval task. Youmans (1991) showed that his text segmentation al- gorithm could be used to manually find scene bound- aries in works of literature. Morris and Hirst (1991) at- *The author would like to thank Christy Doran, Ja- son Eisner, A1 Kim, Mark Liberman, Mitch Marcus, Mike Schultz and David Yarowsky for their helpful comments and acknowledge the support of DARPA grant No. N0014-85- K0018 and ARO grant No. DAAL 03~89-C0031 PRI. tempted to confirm the theories of discourse structure outlined in (Grosz and Sidner, 1986) using information from a thesaurus. In addition, Kozima (1993) specu- lated that segmenting text along topic boundaries may be useful for anaphora resolution and text summariza- tion. This paper is about an automatic method of finding discourse boundaries based on the repetition of lexi- cal items. Halliday and Hasan (1976) and others have claimed that the repetition of lexical items, and in par- ticular content-carrying lexical items, provides coher- ence to a text. This observation has been used implic- itly in several of the techniques described above, but the method presented here depends exclusively on it. Methodology Church (1993) describes a graphical method, called doL- plotting, for aligning bilingual corpora. This method has been adapted here for finding discourse boundaries. The dotplot used for discovering topic boundaries is cre- ated by enumerating the lexical items in an article and plotting points which correspond to word repetitions. …

Content truncated to 500 words