A Computationally Efficient Algorithm for Learning Topical Collocation Models

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1460–1469, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics A Computationally Efficient Algorithm for Learning Topical Collocation Models Zhendong Zhao1, Lan Du1, Benjamin B¨orschinger1,2, John K Pate1, Massimiliano Ciaramita2, Mark Steedman3 and Mark Johnson1 1 Department of Computing, Macquarie University, Australia 2 Google, Zurich, Switzerland 3 School of Informatics, University of Edinburgh, Scotland Abstract Most existing topic models make the bag- of-words assumption that words are gener- ated independently, and so ignore poten- tially useful information about word or- der. Previous attempts to use collocations (short sequences of adjacent words) in topic models have either relied on a pipe- line approach, restricted attention to bi- grams, or resulted in models whose infer- ence does not scale to large corpora. This paper studies how to simultaneously learn both collocations and their topic assign- ments. We present an efficient reformula- tion of the Adaptor Grammar-based topi- cal collocation model (AG-colloc) (John- son, 2010), and develop a point-wise sam- pling algorithm for posterior inference in this new formulation. We further improve the efficiency of the sampling algorithm by exploiting sparsity and parallelising in- ference. Experimental results derived in text classification, information retrieval and human evaluation tasks across a range of datasets show that this reformulation scales to hundreds of thousands of docu- ments while maintaining the good perfor- mance of the AG-colloc model. 1 Introduction Probabilistic topic models like Latent Dirichlet Allocation (LDA) (Blei et al., 2003) are com- monly used to study the meaning of text by iden- tifying a set of latent topics from a collection of documents and assigning each word in these doc- uments to one of the latent topics. A document is modelled as a mixture of latent topics, and each topic is a distribution over a finite vocabulary of words. It is common for topic models to treat documents as bags-of-words, ignoring any inter- nal structure. While this simplifies posterior infer- ence, it also ignores the information encoded in, for example, syntactic relationships (Boyd-Graber and Blei, 2009), word order (Wallach, 2006) and the topic structure of documents (Du et al., 2013). Here we are interested in topic models that capture dependencies between adjacent words in a topic dependent way. For example, the phrase “white house” can be interpreted compositionally in a real-estate context, but not in a political context. Several extensions of LDA have been proposed that assign topics not only to individual words but also to multi-word phrases, which we call topical collocations. However, as we will discuss in sec- tion 2, most of those extensions either rely on a pre-processing step to identify potential colloca- tions (e.g., bigrams and trigrams) or limit attention to bigram dependencies. We want a model that can jointly learn collocations of arbitrary length and their corresponding topic assignments from a large collection of documents. The AG-colloc model (Johnson, 2010) does exactly this. However, be- cause …

Content truncated to 500 words