Textual Information Segmentation by Cohesive Ties

TEXTUAL INFORMATION SEGMENTATION BY COHESIVE TIES Samuel WK Chan, Benjamin K. Thou and CF Choy Language Information Sciences Research Centre City University of Hong Kong Hong Kong ABSTRACT This paper proposes a novel approach in clustering texts automatically into coherent segments. A set of mutual linguistic constraints that largely determines the similarity of meaning among lexical items is used and a weight function is devised to incorporate the diversity of linguistic bonds among the text. A computational method of extracting the gist from a higher order structure representing the tremendous diversity of interrelationship among items is presented. Topic boundaries between segments in a text are identified. Our text segmentation is regarded as a process of identifying the shifts from one segment cluster to another. The experimental results show that the combination of these constraints is capable to address the topic shifts of texts. 1. INTRODUCTION Recent research in textual information science has increasingly turned to text processing beyond sentence level, partly because text analysis is manifestly necessary, and partly through implicit or explicit endorsement that negotiation of meaning in verbal transactions is achieved within the framework of text. Text has a rich structure in which sentences are grouped and related to one another in a variety of ways. A text is usually taken to be a piece of connected and meaningful sentences, rather than just a collection or sequence of sentences, connected only by contiguity. It is clear that text cannot simply be defined as language above the sentence (Winograd, 1972:65). Nor can we assume that stretches of language use which are isomorphic with sentences are the sole business of the grammarian, for these too are realizations of the text process. To understand a text, one must understand the relations between its parts and determine how the various pieces fit together. Clearly, these parts are not just the individual sentences: rather, sentences are joined together to form larger units, which in their turn may be the building blocks of yet larger units. In information science, there is much ongoing research in finding textual regularities on the basis of empirical investigations. Analyzing text structure often calls for more than understanding individual sentences; it usually involves a process of making connections between sentences and sustaining that process as the text progresses (Grosz & Sidner, 1986). Current work also shows that centrality of text should no longer be defined in terms of any simplistic linguistic rules, but rather in terms of linguistic ties which exist among text segments (Stoddard, 1991). Among other kinds of interrelationship which a text may exhibit, cohesion has a prominent role in the understanding of its structure. As advocated by Halliday and Hasan (1976), cohesion can be a unity-creating device in texts. Lexical cohesion, which is a special kind of cohesion, investigates the repetitive sequences of lexically linked (co-articulated) items and their relations to the core sentences. Lexical cohesion has been identified as an important feature which underlines the structure of a wholesome text, distinguishing it from a non-text. …

Content truncated to 500 words