Global Models of Document Structure using Latent Permutations

Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the ACL, pages 371–379, Boulder, Colorado, June 2009. c⃝2009 Association for Computational Linguistics Global Models of Document Structure Using Latent Permutations Harr Chen, S.R.K. Branavan, Regina Barzilay, David R. Karger Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology {harr, branavan, regina, karger}@csail.mit.edu Abstract We present a novel Bayesian topic model for learning discourse-level document structure. Our model leverages insights from discourse theory to constrain latent topic assignments in a way that reflects the underlying organiza- tion of document topics. We propose a global model in which both topic selection and order- ing are biased to be similar across a collection of related documents. We show that this space of orderings can be elegantly represented us- ing a distribution over permutations called the generalized Mallows model. Our structure- aware approach substantially outperforms al- ternative approaches for cross-document com- parison and single-document segmentation.1 1 Introduction In this paper, we introduce a novel latent topic model for the unsupervised learning of document structure. Traditional topic models assume that topics are ran- domly spread throughout a document, or that the succession of topics in a document is Markovian. In contrast, our approach takes advantage of two important discourse-level properties of text in de- termining topic assignments: first, that each docu- ment follows a progression of nonrecurring coher- ent topics (Halliday and Hasan, 1976); and sec- ond, that documents from the same domain tend to present similar topics, in similar orders (Wray, 2002). We show that a topic model incorporat- ing these long-range dependencies outperforms al- 1Code, data, and annotations used in this work are available at http://groups.csail.mit.edu/rbg/code/mallows/ ternative approaches for segmentation and cross- document comparison. For example, consider a collection of encyclope- dia articles about cities. The first constraint captures the notion that a single topic, such as Architecture, is expressed in a contiguous block within the docu- ment, rather than spread over disconnected sections. The second constraint reflects our intuition that all of these related articles will generally mention some major topics associated with cities, such as History and Culture, and will often exhibit similar topic or- derings, such as placing History before Culture. We present a Bayesian latent topic model over re- lated documents that encodes these discourse con- straints by positing a single distribution over a doc- ument’s entire topic structure. This global view on ordering is able to elegantly encode discourse-level properties that would be difficult to represent using local dependencies, such as those induced by hid- den Markov models. Our model enforces that the same topic does not appear in disconnected portions of the topic sequence. Furthermore, our approach biases toward selecting sequences with similar topic ordering, by modeling a distribution over the space of topic permutations. Learning this ordering distribution is a key tech- nical challenge in our proposed approach. For this purpose, we employ the generalized Mallows model, a permutation distribution that concentrates proba- bility mass on a small set of similar …

Content truncated to 500 words