A Phrase-Discovering Topic Model Using Hierarchical Pitman-Yor Processes

Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 214–222, Jeju Island, Korea, 12–14 July 2012. c⃝2012 Association for Computational Linguistics A Phrase-Discovering Topic Model Using Hierarchical Pitman-Yor Processes Robert V. Lindsey University of Colorado, Boulder robert.lindsey@colorado.edu William P. Headden III Two Cassowaries Inc. headdenw@twocassowaries.com Michael J. Stipicevic Google Inc. stip@google.com Abstract Topic models traditionally rely on the bag- of-words assumption. In data mining appli- cations, this often results in end-users being presented with inscrutable lists of topical un- igrams, single words inferred as representa- tive of their topics. In this article, we present a hierarchical generative probabilistic model of topical phrases. The model simultane- ously infers the location, length, and topic of phrases within a corpus and relaxes the bag- of-words assumption within phrases by using a hierarchy of Pitman-Yor processes. We use Markov chain Monte Carlo techniques for ap- proximate inference in the model and perform slice sampling to learn its hyperparameters. We show via an experiment on human subjects that our model finds substantially better, more interpretable topical phrases than do compet- ing models. 1 Introduction Probabilistic topic models have been the focus of intense study in recent years. The archetypal topic model, Latent Dirichlet Allocation (LDA), posits that words within a document are conditionally independent given their topic (Blei et al., 2003). This “bag-of-words” assumption is a common sim- plification in which word order is ignored, but one which introduces undesirable properties into a model meant to serve as an unsupervised ex- ploratory tool for data analysis. When an end-user runs a topic model, the output he or she is often interested in is a list of topical unigrams, words probable in a topic (hence, repre- sentative of it). In many situations, such as during the use of the topic model for the analysis of a new or ill-understood corpus, these lists can be insuffi- ciently informative. For instance, if a layperson ran LDA on the NIPS corpus, he would likely get a topic whose most prominent words include policy, value, and reward. Seeing these words isolated from their context in a list would not be particularly insightful to the layperson unfamiliar with computer science research. An alternative to LDA which produced richer output like policy iteration algorithm, value function, and model-based reinforcement learning alongside the unigrams would be much more en- lightening. Most situations where a topic model is actually useful for data exploration require a model whose output is rich enough to dispel the need for the user’s extensive prior knowledge of the data. Furthermore, lists of topical unigrams are often made only marginally interpretable by virtue of their non-compositionality, the principle that a colloca- tion’s meaning typically is not derivable from its constituent words (Schone and Jurafsky, 2001). For example, the meaning of compact disc as a mu- sic medium comes from neither the unigram com- pact nor the unigram disc, but emerges from the bi- gram as a whole. Moreover, non-compositionality is …

Content truncated to 500 words