Closing the Loop: Fast, Interactive Semi-Supervised Annotation With Queries on Features and Instances

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1467–1478, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics Closing the Loop: Fast, Interactive Semi-Supervised Annotation With Queries on Features and Instances Burr Settles Machine Learning Department Carnegie Mellon University Pittsburgh, PA 15213 USA bsettles@cs.cmu.edu Abstract This paper describes DUALIST, an active learning annotation paradigm which solicits and learns from labels on both features (e.g., words) and instances (e.g., documents). We present a novel semi-supervised training al- gorithm developed for this setting, which is (1) fast enough to support real-time interac- tive speeds, and (2) at least as accurate as pre- existing methods for learning with mixed fea- ture and instance labels. Human annotators in user studies were able to produce near-state- of-the-art classifiers—on several corpora in a variety of application domains—with only a few minutes of effort. 1 Introduction In active learning, a classifier participates in its own training process by posing queries, such as request- ing labels for documents in a text classification task. The goal is to maximize the accuracy of the trained system in the most economically efficient way. This paradigm is well-motivated for natural language ap- plications, where unlabeled data may be readily available (e.g., text on the Internet), but the anno- tation process can be slow and expensive. Nearly all previous work in active learning, how- ever, has focused on selecting queries from the learner’s perspective. For example, experiments are often run in simulation rather than with user stud- ies, and results are routinely evaluated in terms of training set size rather than human annotation time or labor costs (which are more reasonable measures of labeling effort). Many state-of-the-art algorithms are also too slow to run or too tedious to implement to be useful for real-time interaction with human an- notators, and few analyses have taken these factors into account. Furthermore, there is very little work on actively soliciting domain knowledge from hu- mans (e.g., information about features) and incorpo- rating this into the learning process. While selecting good queries is clearly important, if our goal is to reduce actual annotation effort these human factors must be taken into account. In this work, we propose a new interactive annotation inter- face which addresses some of these issues; in partic- ular it has the ability to pose queries on both features (e.g., words) and instances (e.g., documents). We present a novel semi-supervised learning algorithm that is fast, flexible, and accurate enough to support these interface design constraints interactively. 2 DUALIST: Utility for Active Learning with Instances and Semantic Terms Figure 1 shows a screenshot of the DUALIST an- notation tool, which is freely available as an open- source software project1. On the left panel, users are presented with unlabeled documents: in this case Usenet messages that belong to one of two sports- related topics: baseball and hockey. Users may label documents by clicking on the class buttons listed be- low each text. In cases of extreme ambiguity, users may …

Content truncated to 500 words