Active Learning with Efficient Feature Weighting Methods for Improving Data Quality and Classification Accuracy

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 1104–1112, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Active Learning with Efficient Feature Weighting Methods for Improving Data Quality and Classification Accuracy Justin Martineau1, Lu Chen2∗, Doreen Cheng3, and Amit Sheth4 1,3 Samsung Research America, Silicon Valley 1,3 75 W Plumeria Dr. San Jose, CA 95134 USA 2,4 Kno.e.sis Center, Wright State University 2,4 3640 Colonel Glenn Hwy. Fairborn, OH 45435 USA 1,3 {justin.m, doreen.c}@samsung.com 2,4 {chen, amit}@knoesis.org Abstract Many machine learning datasets are noisy with a substantial number of mislabeled instances. This noise yields sub-optimal classification performance. In this paper we study a large, low quality annotated dataset, created quickly and cheaply us- ing Amazon Mechanical Turk to crowd- source annotations. We describe compu- tationally cheap feature weighting tech- niques and a novel non-linear distribution spreading algorithm that can be used to it- eratively and interactively correcting mis- labeled instances to significantly improve annotation quality at low cost. Eight dif- ferent emotion extraction experiments on Twitter data demonstrate that our approach is just as effective as more computation- ally expensive techniques. Our techniques save a considerable amount of time. 1 Introduction Supervised classification algorithms require anno- tated data to teach the machine, by example, how to perform a specific task. There are generally two ways to collect annotations of a dataset: through a few expert annotators, or through crowdsourc- ing services (e.g., Amazon’s Mechanical Turk). High-quality annotations can be produced by ex- pert annotators, but the process is usually slow and costly. The latter option is appealing since it creates a large annotated dataset at low cost. In recent years, there have been an increasing num- ber of studies (Su et al., 2007; Kittur et al., 2008; Sheng et al., 2008; Snow et al., 2008; Callison- Burch, 2009) using crowdsourcing for data anno- tation. However, because annotators that are re- cruited this way may lack expertise and motiva- tion, the annotations tend to be more noisy and ∗This author’s research was done during an internship with Samsung Research America. unreliable, which significantly reduces the perfor- mance of the classification model. This is a chal- lenge faced by many real world applications – given a large, quickly and cheaply created, low quality annotated dataset, how can one improve its quality and learn an accurate classifier from it? Re-annotating the whole dataset is too expen- sive. To reduce the annotation effort, it is desirable to have an algorithm that selects the most likely mislabeled examples first for re-labeling. The pro- cess of selecting and re-labeling data points can be conducted with multiple rounds to iteratively im- prove the data quality. This is similar to the strat- egy of active learning. The basic idea of active learning is to learn an accurate classifier using less training data. An active learner uses a small set of labeled data to iteratively select the most informa- tive instances from a large pool of unlabeled data for human annotators …

Content truncated to 500 words