Low-dimensional Embeddings for Interpretable Anchor-based Topic Inference

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1319–1328, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Low-dimensional Embeddings for Interpretable Anchor-based Topic Inference Moontae Lee Dept. of Computer Science Cornell University Ithaca, NY, 14853 moontae@cs.cornell.edu David Mimno Dept. of Information Science Cornell University Ithaca, NY, 14853 mimno@cornell.edu Abstract The anchor words algorithm performs provably efficient topic model inference by finding an approximate convex hull in a high-dimensional word co-occurrence space. However, the existing greedy al- gorithm often selects poor anchor words, reducing topic quality and interpretability. Rather than finding an approximate con- vex hull in a high-dimensional space, we propose to find an exact convex hull in a visualizable 2- or 3-dimensional space. Such low-dimensional embeddings both improve topics and clearly show users why the algorithm selects certain words. 1 Introduction Statistical topic modeling is useful in exploratory data analysis (Blei et al., 2003), but model infer- ence is known to be NP-hard even for the sim- plest models with only two topics (Sontag and Roy, 2011), and training often remains a black box to users. Likelihood-based training requires expensive approximate inference such as varia- tional methods (Blei et al., 2003), which are deter- ministic but sensitive to initialization, or Markov chain Monte Carlo (MCMC) methods (Griffiths and Steyvers, 2004), which have no finite conver- gence guarantees. Recently Arora et al. proposed the Anchor Words algorithm (Arora et al., 2013), which casts topic inference as statistical recovery using a separability assumption: each topic has a specific anchor word that appears only in the context of that single topic. Each anchor word can be used as a unique pivot to disambiguate the corresponding topic distribution. We then recon- struct the word co-occurrence pattern of each non- anchor words as a convex combination of the co- occurrence patterns of the anchor words. burger salad pizza chicken good told popcorn stadium views tire movies screen sashimi car called hotel yoga bagels shopping dog movie Figure 1: 2D t-SNE projection of a Yelp review corpus and its convex hull. The words corre- sponding to vertices are anchor words for topics, whereas non-anchor words correspond to the inte- rior points. This algorithm is fast, requiring only one pass through the training documents, and provides provable guarantees, but results depend entirely on selecting good anchor words. (Arora et al., 2013) propose a greedy method that finds an approxi- mate convex hull around a set of vectors corre- sponding to the word co-occurrence patterns for each vocabulary word. Although this method is an improvement over previous work that used im- practical linear programming methods (Arora et al., 2012), serious problems remain. The method greedily chooses the farthest point from the cur- rent subspace until the given number of anchors have been found. Particularly at the early stages 1319 of the algorithm, the words associated with the farthest points are likely to be infrequent and id- iosyncratic, and thus form poor bases for human interpretation and topic recovery. This …

Content truncated to 500 words