ALTO: Active Learning with Topic Overviews for Speeding Label Induction and Document Labeling

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1158–1169, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics ALTO: Active Learning with Topic Overviews for Speeding Label Induction and Document Labeling Forough Poursabzi-Sangdeh Computer Science University of Colorado forough.poursabzisangdeh@colorado.edu Jordan Boyd-Graber Computer Science University of Colorado Jordan.Boyd.Graber@colorado.edu Leah Findlater iSchool and UMIACS University of Maryland leahkf@umd.edu Kevin Seppi Computer Science Brigham Young University kseppi@cs.byu.edu Abstract Effective text classification requires experts to annotate data with labels; these training data are time-consuming and expensive to obtain. If you know what labels you want, active learning can reduce the number of labeled documents needed. However, estab- lishing the label set remains difficult. An- notators often lack the global knowledge needed to induce a label set. We intro- duce ALTO: Active Learning with Topic Overviews, an interactive system to help humans annotate documents: topic mod- els provide a global overview of what la- bels to create and active learning directs them to the right documents to label. Our forty-annotator user study shows that while active learning alone is best in extremely resource limited conditions, topic models (even by themselves) lead to better label sets, and ALTO’s combination is best over- all. 1 Introduction Many fields depend on texts labeled by human ex- perts; computational linguistics uses such annota- tion to determine word senses and sentiment (Kelly and Stone, 1975; Kim and Hovy, 2004); while so- cial science uses “coding” to scale up and systeme- tize content analysis (Budge, 2001; Klingemann et al., 2006). Classification takes these labeled data as a train- ing set and labels new data automatically. Creat- ing a broadly applicable and consistent label set that generalizes well is time-consuming and dif- ficult, requiring expensive annotators to examine large swaths of the data. Effective NLP systems must measure (Hwa, 2004; Osborne and Baldridge, 2004; Ngai and Yarowsky, 2000) and reduce an- notation cost (Tomanek et al., 2007). Annotation is hard because it requires both global and local knowledge of the entire dataset. Global knowledge is required to create the set of labels, and local knowledge is required to annotate the most useful examples to serve as a training set for an automatic classifier. The former’s cost is often hidden in mul- tiple rounds of refining annotation guidelines. We create a single interface—ALTO (Active Learning with Topic Overviews)—to address both global and local challenges using two machine learning tools: topic models and active learning (we review both in Section 2). Topic models ad- dress the need for annotators to have a global overview of the data, exposing the broad themes of the corpus so annotators know what labels to cre- ate. Active learning selects documents that help the classifier understand the differences between labels and directs the user’s attention locally to them. We provide users four experimental conditions to com- pare the usefulness of a topic model or a simple list of documents, with or without active learning suggestions (Section 3). We then describe our data and …

Content truncated to 500 words