Text Segmentation Based on Similarity Between Words

TEXT SEGMENTATION BASED ON SIMILARITY BETWEEN WORDS Hideki Kozima Course in Computer Science and Information Mathematics, Graduate School, University of Electro-Communications 1-5-1, Chofugaoka, Chofu, Tokyo 182, Japan (xkozima@phaeton. cs.uec, ac. jp) Abstract This paper proposes a new indicator of text struc- ture, called the lexical cohesion profile (LCP), which locates segment boundaries in a text. A text segment is a coherent scene; the words in a segment a~e linked together via lexical cohesion relations. LCP records mutual similarity of words in a sequence of text. The similarity of words, which represents their cohesiveness, is computed using a semantic network. Comparison with the text segments marked by a number of subjects shows that LCP closely correlates with the hu- man judgments. LCP may provide valuable in- formation for resolving anaphora and ellipsis. INTRODUCTION A text is not just a sequence of words, but it has coherent structure. The meaning of each word can not be determined until it is placed in the structure of the text. Recognizing the structure of text is an essential task in text understanding, especially in resolving anaphora and ellipsis. One of the constituents of the text struc- ture is a text segment. A text segment, whether or not it is explicitly marked, as are sentences and paragraphs, is defined as a sequence of clauses or sentences that display local coherence. It resem- bles a scene in a movie, which describes the same objects in the same situation. This paper proposes an indicator, called the lexical cohesion profile (LCP), which locates seg- ment boundaries in a narrative text. LCP is a record of lexical cohesiveness of words in a se- quence of text. Lexical cohesiveness is defined as word similarity (Kozima and Furugori, 1993) computed by spreading activation on a semantic network. Hills and valleys of LCP closely corre- late with changing of segments. SEGMENTS AND COHERENCE Several methods to capture segment boundaries have been proposed in the studies of text struc- ture. For example, cue phrases play an impor- tant role in signaling segment changes. (Grosz and Sidner, 1986) However, such clues are not di- rectly based on coherence which forms the clauses or sentences into a segment. Youmans (1991) proposed VMP (vocabu- lary management profile) as an indicator of seg- ment boundaries. VMP is a record of the number of new vocabulary terms introduced in an inter- val of text. However, VMP does not work well on a high-density text. The reason is that coherence of a segment should be determined not only by reiteration of words but also by lexical cohesion. Morris and Hirst (1991) used Roget's the- saurus to determine whether or not two words have lexical cohesion. Their method can cap- ture ahnost all the types of lexical cohesion, e.g. systematic and non-systematic semantic relation. However it does not deal with strength of cohe- siveness which suggests the degree of contribution to coherence of the segment. Computing Lexieal Cohesion Kozima and Furugori (1993) defined lexical co- hesiveness as semantic similarity between words, …

Content truncated to 500 words