Fast Inference for Interactive Models of Text

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2997–3006, Osaka, Japan, December 11-17 2016. Fast Inference for Interactive Models of Text Jeffrey Lund, Paul Felt, Kevin Seppi, Eric K. Ringger Department of Computer Science Brigham Young University {jefflund, paul felt, kseppi}@byu.edu, ringger@cs.byu.edu Abstract Probabilistic models are a useful means for analyzing large text corpora. Integrating such models with human interaction enables many new use cases. However, adding human interaction to probabilistic models requires inference algorithms which are both fast and accurate. We explore the use of Iterated Conditional Modes as a fast alternative to Gibbs sampling or variational EM. We demonstrate superior performance both in run time and model quality on three different models of text including a DP Mixture of Multinomials for web search result clustering, the Interactive Topic Model, and MOMRESP, a multinomial crowdsourcing model. 1 Introduction One of the most popular and useful approaches for analysis of large bodies of text documents is proba- bilistic models. For example, topic models such as Latent Dirichlet Allocation (LDA) can automatically learn topics from a set of documents, giving users a glimpse into the common themes of the data (Blei et al., 2003). Other models such as the Mixture of Multinomials can be used to perform document cluster- ing allowing users to automatically organize text data (Meila and Heckerman, 2001; Walker and Ringger, 2008). We are interested in use cases for probabilistic models of text which include human interaction. For example, the Interactive Topic Model (ITM) is a topic model that extends LDA to allow the user to inject model constraints in the form of word groupings while the topics are being learned (Hu et al., 2011). By including the user in the training process rather than simply learning the topics offline, the user can fine- tune the resulting topic model to better suit individual user needs and to accommodate a user’s domain knowledge. However, if the training algorithm is too slow, the delay between receiving user feedback and presenting the updated model will harm the interaction due to increased cognitive load. Consequently, we require an inference algorithm which is both fast enough to facilitate interaction, and maintains (or improves upon) the accuracy of existing inference techniques. For models like LDA, we typically perform training by calculating maximum a posteriori estimates of the latent topic variables and parameters given observed document data, with the idea that the setting of topic variables and parameters which maximizes the posterior distribution will best explain the observed data. Although various exact methods exist, such as belief propagation (Pearl, 1988) and the junction tree algorithm (Koller and Friedman, 2009), the complexity of exact posterior inference on such mod- els is NP-HARD in general, so we resort to various approximations in order to optimize the posterior distribution (Sontag and Roy, 2009; Cooper, 1990). Some popular algorithms for approximate posterior inference include Gibbs sampling and mean field variational inference. Each of these approximate inference algorithms has some drawbacks. For example, while …

Content truncated to 500 words