Structural Topic Model for Latent Topical Structure Analysis

Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 1526–1535, Portland, Oregon, June 19-24, 2011. c⃝2011 Association for Computational Linguistics Structural Topic Model for Latent Topical Structure Analysis Hongning Wang, Duo Zhang, ChengXiang Zhai Department of Computer Science University of Illinois at Urbana-Champaign Urbana IL, 61801 USA {wang296, dzhang22, czhai}@cs.uiuc.edu Abstract Topic models have been successfully applied to many document analysis tasks to discover topics embedded in text. However, existing topic models generally cannot capture the la- tent topical structures in documents. Since languages are intrinsically cohesive and coher- ent, modeling and discovering latent topical transition structures within documents would be beneficial for many text analysis tasks. In this work, we propose a new topic model, Structural Topic Model, which simultaneously discovers topics and reveals the latent topi- cal structures in text through explicitly model- ing topical transitions with a latent first-order Markov chain. Experiment results show that the proposed Structural Topic Model can ef- fectively discover topical structures in text, and the identified structures significantly im- prove the performance of tasks such as sen- tence annotation and sentence ordering. 1 Introduction A great amount of effort has recently been made in applying statistical topic models (Hofmann, 1999; Blei et al., 2003) to explore word co-occurrence pat- terns, i.e. topics, embedded in documents. Topic models have become important building blocks of many interesting applications (see e.g., (Blei and Jordan, 2003; Blei and Lafferty, 2007; Mei et al., 2007; Lu and Zhai, 2008)). In general, topic models can discover word clus- tering patterns in documents and project each doc- ument to a latent topic space formed by such word clusters. However, the topical structure in a docu- ment, i.e., the internal dependency between the top- ics, is generally not captured due to the exchange- ability assumption (Blei et al., 2003), i.e., the doc- ument generation probabilities are invariant to con- tent permutation. In reality, natural language text rarely consists of isolated, unrelated sentences, but rather collocated, structured and coherent groups of sentences (Hovy, 1993). Ignoring such latent topi- cal structures inside the documents means wasting valuable clues about topics and thus would lead to non-optimal topic modeling. Taking apartment rental advertisements as an ex- ample, when people write advertisements for their apartments, it’s natural to first introduce “size” and “address” of the apartment, and then “rent” and “contact”. Few people would talk about “restric- tion” first. If this kind of topical structures are cap- tured by a topic model, it would not only improve the topic mining results, but, more importantly, also help many other document analysis tasks, such as sentence annotation and sentence ordering. Nevertheless, very few existing topic models at- tempted to model such structural dependency among topics. The Aspect HMM model introduced in (Blei and Moreno, 2001) combines pLSA (Hof- mann, 1999) with HMM (Rabiner, 1989) to perform document segmentation over text streams. However, Aspect HMM separately estimates the topics in the training set and depends on heuristics to infer the transitional relations between …

Content truncated to 500 words