Online Multilingual Topic Models with Multi-Level Hyperpriors

Proceedings of NAACL-HLT 2016, pages 454–459, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Online Multilingual Topic Models with Multi-Level Hyperpriors Kriste Krstovski†,§, David A. Smith‡ and Michael J. Kurtz † †Harvard-Smithsonian Center for Astrophysics, Cambridge, MA §College of Information and Computer Sciences, University of Massachusetts Amherst, Amherst, MA ‡College of Computer and Information Science, Northeastern University, Boston, MA kkrstovski@cfa.harvard.edu, dasmith@ccs.neu.edu, kurtz@cfa.harvard.edu Abstract For topic models, such as LDA, that use a bag-of-words assumption, it becomes es- pecially important to break the corpus into appropriately-sized “documents”. Since the models are estimated solely from the term cooccurrences, extensive documents such as books or long journal articles lead to diffuse statistics, and short documents such as forum posts or product reviews can lead to sparsity. This paper describes practical inference pro- cedures for hierarchical models that smooth topic estimates for smaller sections with hy- perpriors over larger documents. Importantly for large collections, these online variational Bayes inference methods perform a single pass over a corpus and achieve better perplex- ity than “flat” topic models on monolingual and multilingual data. Furthermore, on the task of detecting document translation pairs in large multilingual collections, polylingual topic models (PLTM) with multi-level hyper- priors (mlhPLTM) achieve significantly better performance than existing online PLTM mod- els while retaining computational efficiency. 1 Introduction Bag of words models simplify the representation of documents by discarding grammatical information and simply relying on document-level word cooc- currence statistics. Topic models, such as latent Dirichlet allocation (LDA) (Blei et al., 2003), use this representation. A major drawback of the bag of words representation, especially in collections of large documents, is that the word co-occurrence statistics are computed on a document level and as such they do not capture the effect of words co-occurring close to each other versus words co- occurring further apart. One alternative approach to longer documents that has received attention in the past has been to directly model local—i.e., Markov—dependencies among tokens. For example, the topical n-gram model (TNG) introduced by Wang et al. (2007) models unigram and n-gram phrases as mixture of topics based on the nearby word context. More recently, Jameel & Lam (2013) proposed an LDA extension that uses word sequence information to generate topic distribution over n-grams and per- forms topic segmentation using segment and para- graph information. While these and many other ap- proaches offer a better and more realistic modeling of word sequences, they don’t model topical varia- tions across document sections either in mono- or multilingual collections. In this paper, we focus on hierarchical models for improving topic models of long documents. In the past, document-topic based hierarchical prior struc- tures have been explored for LDA. For example, Wallach et al. (2009) showed that Gibbs sampling implementation of asymmetric Dirichlet priors pro- vide better modeling of documents, across the whole collection, compared to the original LDA approach. More recently, Kim et al. (2013) introduced tiLDA, a topic model of monolingual document collections with nested hierarchies. In order to achieve …

Content truncated to 500 words