Evaluating Visual Representations for Topic Understanding and Their Effects on Manually Generated Topic Labels

Evaluating Visual Representations for Topic Understanding and Their Effects on Manually Generated Topic Labels Alison Smith∗ Tak Yeon Lee∗ Forough Poursabzi-Sangdeh† Jordan Boyd-Graber† Niklas Elmqvist∗ Leah Findlater∗ ∗University of Maryland, College Park, MD †University of Colorado, Boulder, CO {amsmit,tylee}@cs.umd.edu {forough.poursabzisangdeh, jordan.boyd.graber}@colorado.edu {elm,leahkf}@cs.umd.edu Abstract Probabilistic topic models are important tools for indexing, summarizing, and analyzing large document collections by their themes. However, promoting end-user understanding of topics remains an open research prob- lem. We compare labels generated by users given four topic visualization techniques— word lists, word lists with bars, word clouds, and network graphs—against each other and against automatically generated labels. Our basis of comparison is participant ratings of how well labels describe documents from the topic. Our study has two phases: a label- ing phase where participants label visualized topics and a validation phase where different participants select which labels best describe the topics’ documents. Although all visual- izations produce similar quality labels, sim- ple visualizations such as word lists allow par- ticipants to quickly understand topics, while complex visualizations take longer but expose multi-word expressions that simpler visualiza- tions obscure. Automatic labels lag behind user-created labels, but our dataset of man- ually labeled topics highlights linguistic pat- terns (e.g., hypernyms, phrases) that can be used to improve automatic topic labeling al- gorithms. 1 Comprehensible Topic Models Needed A central challenge of the “big data” era is to help users make sense of large text collections (Hotho et al., 2005). A common approach to summarizing the main themes in a corpus is to use topic models (Blei, 2012), which are data-driven statistical models that identify words that appear together in similar docu- ments. These sets of words or “topics” evince inter- nal coherence and can help guide users to relevant documents. For instance, an FBI investigator sifting through the released Hillary Clinton e-mails may see a topic with the words “Benghazi”, “Libya”, “Blu- menthal”, and “success”, spurring the investigator to dig deeper to find further evidence of inappro- priate communication with longtime friend Sidney Blumenthal regarding Benghazi. A key challenge for topic modeling, however, is how to promote end-user understanding of individ- ual topics and the overall model. Most existing topic presentations use simple word lists (Chaney and Blei, 2012; Eisenstein et al., 2012). Although a variety of alternative topic visualization techniques exist (Sievert and Shirley, 2014; Yi et al., 2005), there has been no systematic assessment to compare them. Beyond exploring different visualization tech- niques, another means of making topics easier for users to understand is to provide descriptive labels to complement a topic’s set of words (Aletras et al., 2014). Unfortunately, manual labeling is slow and, while automatic labeling approaches exist (Lau et al., 2010; Mei et al., 2007; Lau et al., 2011), their effectiveness is not guaranteed for all tasks. To better understand these problems, we use la- beling to evaluate topic model visualizations. Our study compares the impact of four commonly used topic visualization techniques on the labels that users create when interpreting a topic …

Content truncated to 500 words