Using Collocations for Topic Segmentation and Link Detection

Using collocations for topic segmentation and link detection Olivier FERRET CEA – LIST/LIC2M Route du Panorama – BP 6 92265 Fontenay-aux-Roses Cedex olivier.ferret@cea.fr Abstract We present in this paper a method for achiev- ing in an integrated way two tasks of topic analysis: segmentation and link detection. This method combines word repetition and the lexi- cal cohesion stated by a collocation network to compensate for the respective weaknesses of the two approaches. We report an evaluation of our method for segmentation on two cor- pora, one in French and one in English, and we propose an evaluation measure that spe- cifically suits that kind of systems. 1 Introduction Topic analysis, which aims at identifying the top- ics of a text, delimiting their extend and finding the relations between the resulting segments, has recently raised an important interest. The largest part of it was dedicated to topic segmentation, also called linear text segmentation, and to the TDT (Topic Detection and Tracking) initiative (Fiscus et al., 1999), which addresses all the tasks we have mentioned but from a domain-dependent view- point and not necessarily in an integrated way. Systems that implement this work can be catego- rized according to what kind of knowledge they use. Most of those that achieve text segmentation only rely on the intrinsic characteristics of texts: word distribution, as in (Hearst, 1997), (Choi, 2000) and (Utiyama and Isahara, 2001), or lin- guistic cues as in (Passonneau and Litman, 1997). They can be applied without restriction about do- mains but have low results when a text doesn’t characterize its topical structure by surface clues. Some systems exploit domain-independent knowl- edge about lexical cohesion: a network of words built from a dictionary in (Kozima, 1993); a large set of collocations collected from a corpus in (Fer- ret, 1998), (Kaufmann, 1999) and (Choi, 2001). To some extend, this knowledge permits these sys- tems to discard some false topical shifts without losing their independence with regard to domains. The last main type of systems relies on knowledge about the topics they may encounter in the texts they process. This is typically the kind of approach developed in TDT where this knowledge is auto- matically built from a set of reference texts. The work of Bigi (Bigi et al., 1998) stands in the same perspective but focuses on much larger topics than TDT. These systems have a limited scope due to their topic representations but they are also more precise for the same reason. Hybrid systems that combine the approaches we have presented were also developed and illustrated the interest of such a combination: (Jobbins and Evett, 1998) combined word recurrence, colloca- tions and a thesaurus; (Beeferman et al., 1999) relied on both collocations and linguistic cues. The topic analysis we propose implements such a hybrid approach: it relies on a general language resource, a collocation network, but exploits it together with word recurrence in texts. Moreover, it simultaneously achieves topic segmentation and link detection, i.e. determining whether two seg- ments discuss the …

Content truncated to 500 words