Multi-Paragraph Segmentation Expository Text

MULTI-PARAGRAPH SEGMENTATION EXPOSITORY TEXT Marti A. Hearst Computer Science Division, 571 Evans Hall University of California, Berkeley Berkeley, CA 94720 and Xerox Palo Alto Research Center marti @cs. berkeley, edu OF Abstract This paper describes TextTiling, an algorithm for parti- tioning expository texts into coherent multi-paragraph discourse units which reflect the subtopic structure of the texts. The algorithm uses domain-independent lex- ical frequency and distribution information to recog- nize the interactions of multiple simultaneous themes. Two fully-implemented versions of the algorithm are de- scribed and shown to produce segmentation that corre- sponds well to human judgments of the major subtopic boundaries of thirteen lengthy texts. INTRODUCTION The structure of expository texts can be characterized as a sequence ofsubtopical discussions that occur in the context of a few main topic discussions. For example, a popular science text called Stargazers, whose main topic is the existence of life on earth and other planets, can be described as consisting of the following subdiscussions (numbers indicate paragraph numbers): 1-3 Intro - the search for life in space 4-5 The moon's chemical composition 6-8 How early proximity of the moon shaped it 9-12 How the moon helped life evolve on earth 13 Improbability of the earth-moon system 14-16 Binary/trinary star systems make life un- likely 17-18 The low probability of non-binary/trinary systems 19-20 Properties of our sun that facilitate life 21 Summary Subtopic structure is sometimes marked in techni- cal texts by headings and subheadings which divide the text into coherent segments; Brown & Yule (1983:140) state that this kind of division is one of the most basic in discourse. However, many expository texts consist of long sequences of paragraphs with very little structural demarcation. This paper presents fully-implemented al- gorithms that use lexical cohesion relations to partition expository texts into multi-paragraph segments that re- flect their subtopic structure. Because the model of dis- course structure is one in which text is partitioned into contiguous, nonoverlapping blocks, I call the general approach TextTiling. The ultimate goal is to not only identify the extents of the subtopical units, but to label their contents as well. This paper focusses only on the discovery of subtopic structure, leaving determination of subtopic content to future work. Most discourse segmentation work is done at a finer granularity than that suggested here. However, for lengthy written expository texts, multi-paragraph seg- mentation has many potential uses, including the im- provement of computational tasks that make use of dis- tributional information. For example, disambiguation algorithms that train on arbitrary-size text windows, e.g., Yarowsky (1992) and Gale et ai. (1992b), and al- gorithms that use lexical co-occurrence to determine se- mantic relatedness, e.g., Schfitze (1993), might benefit from using windows with motivated boundaries instead. Information retrieval algorithms can use subtopic structuring to return meaningful portions of a text if paragraphs are too short and sections are too long (or are not present). Motivated segments can also be used as a more meaningful unit for indexing long texts. Salton et al. (1993), working with encyclopedia …

Content truncated to 500 words