Improving Text Segmentation by Combining Endogenous and Exogenous Methods

International Conference RANLP 2009 - Borovets, Bulgaria, pages 88–93 Improving text segmentation by combining endogenous and exogenous methods Olivier Ferret CEA, LIST 18 route du Panorama, BP6, Fontenay-aux-Roses, F-92265 France olivier.ferret@cea.fr Abstract Topic segmentation was addressed by a large amount of work from which it is not easy to draw conclusions, especially about the need for knowl- edge. In this article, we propose to combine in the same framework two methods for improving the results of a topic segmenter based on lexical reiteration. The first one is endogenous and ex- ploits the distributional similarity of words in a document for discovering its topics. These topics are then used to facilitate the detection of topi- cal similarity between discourse units. The sec- ond approach achieves the same goal by relying on external resources. Two resources are tested: a network of lexical co-occurrences built from a large corpus and a set of word senses induced from this network. An evaluation of the two ap- proaches and their combination is performed in a reference framework and shows the interest of this combination both for French and English. 1 Introduction In this article, we address the problem of linear topic segmentation, which consists in segmenting documents into topically homogeneous non-overlapping segments. This Discourse Analysis problem has received a con- stant interest since works such as [11]. One criterion for classifying topic segmentation systems is the kind of knowledge they depend on. Most of them only rely on surface features of documents: word reiteration in [11, 4, 20, 10], and more recently [14, 7], or discourse cues in [16, 10]. As they don’t exploit external knowl- edge, such systems are not domain-dependent but they can be successfully applied only to some types of doc- uments: word reiteration is reliable only if concepts are not expressed by too different means (synonyms, etc.); discourse cues are often rare and corpus-specific. To overcome these difficulties, some systems make use of domain-independent knowledge about lexical cohesion: a lexical network built from a dictionary in [13]; a thesaurus in [15]; a large set of lexical co- occurrences collected from a corpus in [5] or [6]. To some extent, these lexical networks enable segmenters to rely on a sort of concept reiteration. However, their lack of any topical structure makes this kind of knowl- edge difficult to use when lexical ambiguity is high. The most simple solution to this problem is to ex- ploit knowledge about the topics that may occur in documents. Such topic models are generally built from a large set of example documents as in [21], [2] or in one component of [1]. These statistical topic models enable segmenters to improve their precision but they also restrict their scope. Hybrid systems that combine the approaches we have presented were also developed and illustrated the interest of such a combination: [12] combined word re- currence, co-occurrences and a thesaurus; [1] relied on both lexical modeling and discourse cues; [10] made use of word reiteration through lexical chains and dis- …

Content truncated to 500 words