Modelling Sequential Text with an Adaptive Topic Model

Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 535–545, Jeju Island, Korea, 12–14 July 2012. c⃝2012 Association for Computational Linguistics Modelling Sequential Text with an Adaptive Topic Model Lan Du∗ Department of Computing Macquarie University Sydney, Australia lan.du@mq.edu.au Wray Buntine∗ Canberra Research Lab National ICT Australia Canberra, Australia wray.buntine@nicta.com.au Huidong Jin∗ CSIRO Mathematics, Informatics and Statistics, Canberra, Australia warren.jin@csiro.au Abstract Topic models are increasingly being used for text analysis tasks, often times replacing ear- lier semantic techniques such as latent seman- tic analysis. In this paper, we develop a novel adaptive topic model with the ability to adapt topics from both the previous segment and the parent document. For this proposed model, a Gibbs sampler is developed for doing poste- rior inference. Experimental results show that with topic adaptation, our model significantly improves over existing approaches in terms of perplexity, and is able to uncover clear se- quential structure on, for example, Herman Melville’s book “Moby Dick”. 1 Introduction Natural language text usually consists of topically structured and coherent components, such as groups of sentences that form paragraphs and groups of paragraphs that form sections. Topical coherence in documents facilitates readers’ comprehension, and reflects the author’s intended structure. Capturing this structural topical dependency should lead to im- proved topic modelling. It also seems reasonable to propose that text analysis tasks that involve the structure of a document, for instance, summarisation and segmentation, should also be improved by topic models that better model that structure. Recently, topic models are increasingly being used for text analysis tasks such as summarisa- ∗This work was partially done when Du was at College of Engineering & Computer Science, the Australian National Uni- versity when working together with Buntine and Jin there. tion (Arora and Ravindran, 2008) and segmenta- tion (Misra et al., 2011; Eisenstein and Barzilay, 2008), often times replacing earlier semantic tech- niques such as latent semantic analysis (Deerwester et al., 1990). Topic models can be improved by bet- ter modelling the semantic aspects of text, for in- stance integrating collocations into the model (John- son, 2010; Hardisty et al., 2010) or encouraging top- ics to be more semantically coherent (Newman et al., 2011) based on lexical coherence models (New- man et al., 2010), modelling the structural aspects of documents, for instance modelling a document as a set of segments (Du et al., 2010; Wang et al., 2011; Chen et al., 2009), or improving the under- lying statistical methods (Teh et al., 2006; Wallach et al., 2009). Topic models, like statistical parsing methods, are using more sophisticated latent vari- able methods in order to model different aspects of these problems. In this paper, we are interested in developing a new topic model which can take into account the structural topic dependency by following the higher level document subject structure, but we hope to re- tain the general flavour of topic models, where com- ponents (e.g., sentences) can be a mixture of topics. Thus …

Content truncated to 500 words