Active Learning by Labeling Features

Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 81–90, Singapore, 6-7 August 2009. c⃝2009 ACL and AFNLP Active Learning by Labeling Features Gregory Druck Dept. of Computer Science University of Massachusetts Amherst, MA 01003 gdruck@cs.umass.edu Burr Settles Dept. of Biostatistics & Medical Informatics Dept. of Computer Sciences University of Wisconsin Madison, WI 53706 bsettles@cs.wisc.edu Andrew McCallum Dept. of Computer Science University of Massachusetts Amherst, MA 01003 mccallum@cs.umass.edu Abstract Methods that learn from prior informa- tion about input features such as general- ized expectation (GE) have been used to train accurate models with very little ef- fort. In this paper, we propose an ac- tive learning approach in which the ma- chine solicits “labels” on features rather than instances. In both simulated and real user experiments on two sequence label- ing tasks we show that our active learning method outperforms passive learning with features as well as traditional active learn- ing with instances. Preliminary experi- ments suggest that novel interfaces which intelligently solicit labels on multiple fea- tures facilitate more efficient annotation. 1 Introduction The application of machine learning to new prob- lems is slowed by the need for labeled training data. When output variables are structured, an- notation can be particularly difficult and time- consuming. For example, when training a condi- tional random field (Lafferty et al., 2001) to ex- tract fields such as rent, contact, features, and utilities from apartment classifieds, labeling 22 instances (2,540 tokens) provides only 66.1% accuracy.1 Recent work has used unlabeled data and lim- ited prior information about input features to boot- strap accurate structured output models. For ex- ample, both Haghighi and Klein (2006) and Mann and McCallum (2008) have demonstrated results better than 66.1% on the apartments task de- scribed above using only a list of 33 highly dis- criminative features and the labels they indicate. However, these methods have only been applied in scenarios in which the user supplies such prior knowledge before learning begins. 1Averaged over 10 randomly selected sets of 22 instances. In traditional active learning (Settles, 2009), the machine queries the user for only the labels of in- stances that would be most helpful to the machine. This paper proposes an active learning approach in which the user provides “labels” for input features, rather than instances. A labeled input feature de- notes that a particular input feature, for example the word call, is highly indicative of a particular label, such as contact. Table 1 provides an excerpt of a feature active learning session. In this paper, we advocate using generalized expectation (GE) criteria (Mann and McCallum, 2008) for learning with labeled features. We pro- vide an alternate treatment of the GE objective function used by Mann and McCallum (2008) and a novel speedup to the gradient computation. We then provide a pool-based feature active learning algorithm that includes an option to skip queries, for cases in which a feature has no clear label. We propose and evaluate feature query selection algorithms that aim to …

Content truncated to 500 words