Leveraging Lexical Cohesion and Disruption for Topic Segmentation

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1314–1324, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Leveraging lexical cohesion and disruption for topic segmentation Anca S¸imon Universit´e de Rennes 1 IRISA & INRIA Rennes Guillaume Gravier CNRS IRISA & INRIA Rennes anca-roxana.simon@irisa.fr guillaume.gravier@irisa.fr pascale.sebillot@irisa.fr Pascale S´ebillot INSA de Rennes IRISA & INRIA Rennes Abstract Topic segmentation classically relies on one of two criteria, either finding areas with co- herent vocabulary use or detecting discontinu- ities. In this paper, we propose a segmenta- tion criterion combining both lexical cohesion and disruption, enabling a trade-off between the two. We provide the mathematical formu- lation of the criterion and an efficient graph based decoding algorithm for topic segmenta- tion. Experimental results on standard textual data sets and on a more challenging corpus of automatically transcribed broadcast news shows demonstrate the benefit of such a com- bination. Gains were observed in all condi- tions, with segments of either regular or vary- ing length and abrupt or smooth topic shifts. Long segments benefit more than short seg- ments. However the algorithm has proven ro- bust on automatic transcripts with short seg- ments and limited vocabulary reoccurrences. 1 Introduction Topic segmentation consists in evidentiating the se- mantic structure of a document: Algorithms devel- oped for this task aim at automatically detecting frontiers which define topically coherent segments in a text. Various methods for topic segmentation of tex- tual data are described in the literature, e.g., (Rey- nar, 1994; Hearst, 1997; Ferret et al., 1998; Choi, 2000; Moens and Busser, 2001; Utiyama and Isa- hara, 2001), most of them relying on the notion of lexical cohesion, i.e., identifying segments with a consistent use of vocabulary, either based on words or on semantic relations between words. Reoccur- rences of words or related words and lexical chains are two popular methods to evidence lexical cohe- sion. This general principle of lexical cohesion is further exploited for topic segmentation with two radically different strategies. On the one hand, a measure of the lexical cohesion can be used to deter- mine coherent segments (Reynar, 1994; Moens and Busser, 2001; Utiyama and Isahara, 2001). On the other hand, shifts in the use of vocabulary can be searched for to directly identify the segment fron- tiers by measuring the lexical disruption (Hearst, 1997). Techniques based on the first strategy yield more accurate segmentation results, but face a problem of over-segmentation which can, up to now, only be solved by providing prior information regarding the distribution of segment length or the expected num- ber of segments. In this paper, we propose a segmen- tation criterion combining both cohesion and dis- ruption along with the corresponding algorithm for topic segmentation. Such a criterion ensures a co- herent use of vocabulary within each resulting seg- ment, as well as a significant difference of vocabu- lary between neighboring segments. Moreover, the combination of these two strategies enables regular- izing the number of segments found without resort- ing to …

Content truncated to 500 words