Bayesian Unsupervised Topic Segmentation

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 334–343, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics Bayesian Unsupervised Topic Segmentation Jacob Eisenstein and Regina Barzilay Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology 77 Massachusetts Ave., Cambridge MA 02139 {jacobe,regina}@csail.mit.edu Abstract This paper describes a novel Bayesian ap- proach to unsupervised topic segmentation. Unsupervised systems for this task are driven by lexical cohesion: the tendency of well- formed segments to induce a compact and consistent lexical distribution. We show that lexical cohesion can be placed in a Bayesian context by modeling the words in each topic segment as draws from a multinomial lan- guage model associated with the segment; maximizing the observation likelihood in such a model yields a lexically-cohesive segmenta- tion. This contrasts with previous approaches, which relied on hand-crafted cohesion met- rics. The Bayesian framework provides a prin- cipled way to incorporate additional features such as cue phrases, a powerful indicator of discourse structure that has not been previ- ously used in unsupervised segmentation sys- tems. Our model yields consistent improve- ments over an array of state-of-the-art systems on both text and speech datasets. We also show that both an entropy-based analysis and a well-known previous technique can be de- rived as special cases of the Bayesian frame- work.1 1 Introduction Topic segmentation is one of the fundamental prob- lems in discourse analysis, where the task is to divide a text into a linear sequence of topically- coherent segments. Hearst’s TEXTTILING (1994) introduced the idea that unsupervised segmentation 1Code and materials for this work are available at http://groups.csail.mit.edu/rbg/code/ bayesseg/. can be driven by lexical cohesion, as high-quality segmentations feature homogeneous lexical distri- butions within each topic segment. Lexical cohesion has provided the inspiration for several successful systems (e.g., Utiyama and Isahara, 2001; Galley et al.2003; Malioutov and Barzilay, 2006), and is cur- rently the dominant approach to unsupervised topic segmentation. But despite the effectiveness of lexical cohesion for unsupervised topic segmentation, it is clear that there are other important indicators that are ignored by the current generation of unsupervised systems. For example, consider cue phrases, which are ex- plicit discourse markers such as “now” or “how- ever” (Grosz and Sidner, 1986; Hirschberg and Lit- man, 1993; Knott, 1996). Cue phrases have been shown to be a useful feature for supervised topic segmentation (Passonneau and Litman, 1993; Gal- ley et al., 2003), but cannot be incorporated by current unsupervised models. One reason for this is that existing unsupervised methods use arbitrary, hand-crafted metrics for quantifying lexical cohe- sion, such as weighted cosine similarity (Hearst, 1994; Malioutov and Barzilay, 2006). Without su- pervision, it is not possible to combine such met- rics with additional sources of information. More- over, such hand-crafted metrics may not general- ize well across multiple datasets, and often include parameters which must be tuned on development sets (Malioutov and Barzilay, 2006; Galley et al., 2003). In this paper, we situate lexical cohesion in a Bayesian framework, allowing other …

Content truncated to 500 words