Needle in a Haystack: Reducing the Costs of Annotating Rare-Class Instances in Imbalanced Datasets

PACLIC 28 !244 Needle in a Haystack: Reducing the Costs of Annotating Rare-Class Instances in Imbalanced Datasets Emily K. Jamison ‡ and Iryna Gurevych †‡ ‡Ubiquitous Knowledge Processing Lab (UKP-TUDA), Department of Computer Science, Technische Universit¨at Darmstadt † Ubiquitous Knowledge Processing Lab (UKP-DIPF), German Institute for Educational Research http://www.ukp.tu-darmstadt.de Abstract Crowdsourced data annotation is noisier than annotation from trained workers. Previ- ous work has shown that redundant annota- tions can eliminate the agreement gap be- tween crowdsource workers and trained work- ers. Redundant annotation is usually non- problematic because individual crowdsource judgments are inconsequentially cheap in a class-balanced dataset. However, redundant annotation on class- imbalanced datasets requires many more la- bels per instance. In this paper, using three class-imbalanced corpora, we show that an- notation redundancy for noise reduction is very expensive on a class-imbalanced dataset, and should be discarded for instances receiv- ing a single common-class label. We also show that this simple technique produces an- notations at approximately the same cost of a metadata-trained, supervised cascading ma- chine classifier, or about 70% cheaper than 5- vote majority-vote aggregation. 1 Introduction The advent of crowdsourcing as a cheap but noisy source for annotation labels has spurred the devel- opment of algorithms to maximize quality and mini- mize cost. Techniques can detect spammers (Oleson et al., 2011; Downs et al., 2010; Buchholz and La- torre, 2011), model worker quality and bias during label aggregation (Jung and Lease, 2012; Ipeirotis et al., 2010) and optimize obtaining more labels per in- stance or more labelled instances (Kumar and Lease, 2011; Sheng et al., 2008). However, much previ- ous work for quality maximization and cost limita- tion assumes that the dataset to be annotated is class- balanced. Class-imbalanced datasets, or datasets with dif- ferences in prior class probabilities, present a unique problem during corpus production: how to include enough rare-class instances in the corpus to train a machine learner? If the orginal class distribution is maintained, a corpus that is large enough for a machine learner to identify common-class (i.e., fre- quent class) instances may suffer from a lack of rare-class (i.e., infrequent class) instances. Yet, it can be cost-prohibitive to expand the corpus until enough rare-class instances are included. Content-based instance targeting can be used to select instances with a high probability of being rare-class. For example, in a binary class annota- tion task identifying pairs of emails from the same thread, where most instances are negative, cosine text similarity between the emails can be used to identify pairs of emails that are likely to be posi- tive, so that they could be annotated and included in the resulting class-balanced corpus (Jamison and Gurevych, 2013). However, this technique renders the corpus useless for experiments including token similarity (or ngram similarity, semantic similarity, stopword distribution similarity, keyword similarity, etc) as a feature; a machine learner would be likely to learn the very same features for classification that were used to identify the rare-class instances during corpus construction. Even worse, Mikros and Argiri (2007) …

Content truncated to 500 words