TopicCheck: Interactive Alignment for Assessing Topic Model Stability

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 175–184, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics TopicCheck: Interactive Alignment for Assessing Topic Model Stability Jason Chuang∗ jason@chuang.ca Margaret E. Roberts† Political Science U. California, San Diego meroberts@ucsd.edu Brandon M. Stewart† Government Harvard University bstewart@fas.harvard.edu Rebecca Weiss† Communication Stanford University rjweiss@stanford.edu Dustin Tingley Government Harvard University dtingley@gov.harvard.edu Justin Grimmer Political Science Stanford University jgrimmer@stanford.edu Jeffrey Heer Computer Science & Eng. University of Washington jheer@uw.edu Abstract Content analysis, a widely-applied social sci- ence research method, is increasingly being supplemented by topic modeling. However, while the discourse on content analysis cen- ters heavily on reproducibility, computer sci- entists often focus more on scalability and less on coding reliability, leading to growing skep- ticism on the usefulness of topic models for automated content analysis. In response, we introduce TopicCheck, an interactive tool for assessing topic model stability. Our contri- butions are threefold. First, from established guidelines on reproducible content analysis, we distill a set of design requirements on how to computationally assess the stability of an automated coding process. Second, we devise an interactive alignment algorithm for match- ing latent topics from multiple models, and en- able sensitivity evaluation across a large num- ber of models. Finally, we demonstrate that our tool enables social scientists to gain novel insights into three active research questions. 1 Introduction Content analysis — the examination and systematic categorization of written texts (Berelson, 1952) — is a fundamental and widely-applied research method in the social sciences and humanities (Krippendorff, 2004a), found in one third of all articles published in major communication journals (Wimmer and Do- minick, 2010). Initial reading and coding, two labor- ∗Work completed while at Stanford University and the Uni- versity of Washington, and submitted while at the Allen Insti- tute for Artificial Intelligence. †These authors contributed equally to this paper. intensive steps in the analysis process, are increas- ingly replaced by computational approaches such as statistical topic modeling (Grimmer, 2013; McFar- land et al., 2013; Roberts et al., 2014a). However, while the discourse on content analysis overwhelmingly centers around the reproducibility and generalizability of a coding scheme (Krippen- dorff, 2004b; Lombard et al., 2002), computer sci- entists tend to focus more on increasing the scale of analysis and less on establishing coding reliabil- ity. Machine-generated latent topics are often taken on faith to be a truthful and consistent representa- tion of the underlying corpus, but in practice ex- hibit significant variations among models or mod- eling runs. These unquantified uncertainties fuel growing skepticism (Schmidt, 2012) and hamper the continued adoption (Grimmer and Stewart, 2011) of topic models for automated content analysis. In response, we introduce TopicCheck, an interac- tive tool for assessing the stability of topic models. Our threefold contributions are as follows. First, from established guidelines on reproducible content analysis, we distill a set of design require- ments on how to computationally assess the stabil- ity of an automated coding process. …

Content truncated to 500 words