Why Didn’t You Listen to Me? Comparing User Control of Human-in-the-Loop Topic Models

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6323–6330 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6323 Why Didn’t You Listen to Me? Comparing User Control of Human-in-the-Loop Topic Models Varun Kumar ∗ Amazon Alexa Cambridge, MA varunk@cs.umd.edu Alison Smith-Renner University of Maryland College Park, MD amsmit@cs.umd.edu Leah Findlater University of Washington Seattle, Washington leahkf@uw.edu Kevin Seppi Brigham Young University Provo, UT kseppi@gmail.com Jordan Boyd-Graber University of Maryland College Park, MD jordanbg@umiacs.edu Abstract To address the lack of comparative evalua- tion of Human-in-the-Loop Topic Modeling (HLTM) systems, we implement and evaluate three contrasting HLTM modeling approaches using simulation experiments. These ap- proaches extend previously proposed frame- works, including constraints and informed prior-based methods. Users should have a sense of control in HLTM systems, so we pro- pose a control metric to measure whether re- finement operations’ results match users’ ex- pectations. Informed prior-based methods pro- vide better control than constraints, but con- straints yield higher quality topics. 1 Human-in-the-Loop Topic Modeling Topic models help explore large, unstructured text corpora by automatically discovering the topics discussed in the documents (Blei et al., 2003). However, generated topic models are not perfect; they may contain incoherent or loosely connected topics (Chang et al., 2009; Mimno et al., 2011; Boyd-Graber et al., 2014). Human-in-the-Loop Topic Modeling (HLTM) addresses these issues by incorporating human knowledge into the modeling process. Existing HLTM systems expose topic models as their topic words and documents, and users provide feedback to improve the models using varied refinement op- erations, such as adding words to topics, merg- ing topics, or removing documents (Smith et al., 2018; Wang et al., 2019). Systems also vary in how they incorporate feedback, such as “must- ∗Work performed at University of Maryland, College Park link” and “cannot-link” constraints (Andrzejew- ski et al., 2009; Hu et al., 2014), informed pri- ors (Smith et al., 2018), or document labels (Yang et al., 2015). However, evaluations of these sys- tems are either not comparative (Choo et al., 2013; Lee et al., 2017) or compare against non- interactive models (Hoque and Carenini, 2015; Hu et al., 2014) or for only a limited set of refine- ments (Yang et al., 2015; Xie et al., 2015). Evalu- ations are thus silent on which HLTM system best supports users in improving topic models: they ig- nore whether refinements are applied correctly or how they compare with other approaches. More- over, comparative evaluations can be difficult be- cause existing HLTM systems support diverse re- finement operations with little overlap. To address these issues, we implement three HLTM systems that differ in the techniques for incorporating prior knowledge (informed priors vs. constraints) and for inference (Gibbs sampling vs. variational EM), but that all support seven re- finement operations preferred by end users (Lee et al., 2017; Musialek et al., 2016). We compare these systems through experiments simulating ran- dom and “good” user behavior. The two Gibbs sampling-based systems extend prior work (Yang et al., …

Content truncated to 500 words