Assessing Benefit from Feature Feedback in Active Learning for Text Classification

Proceedings of the Fifteenth Conference on Computational Natural Language Learning, pages 106–114, Portland, Oregon, USA, 23–24 June 2011. c⃝2011 Association for Computational Linguistics Assessing Benefit from Feature Feedback in Active Learning for Text Classification Shilpa Arora Language Technologies Institute School of Computer Science Carnegie Mellon University shilpaa@cs.cmu.edu Eric Nyberg Language Technologies Institute School of Computer Science Carnegie Mellon University ehn@cs.cmu.edu Abstract Feature feedback is an alternative to instance labeling when seeking supervision from hu- man experts. Combination of instance and feature feedback has been shown to reduce the total annotation cost for supervised learning. However, learning problems may not benefit equally from feature feedback. It is well un- derstood that the benefit from feature feed- back reduces as the amount of training data increases. We show that other characteristics such as domain, instance granularity, feature space, instance selection strategy and propor- tion of relevant text, have a significant effect on benefit from feature feedback. We estimate the maximum benefit feature feedback may provide; our estimate does not depend on how the feedback is solicited and incorporated into the model. We extend the complexity mea- sures proposed in the literature and propose some new ones to categorize learning prob- lems, and find that they are strong indicators of the benefit from feature feedback. 1 Introduction Linear classifiers model the response as a weighted linear combination of the features in input instances. A supervised approach to learning a linear classifier involves learning the weights for the features from labeled data. A large number of labeled instances may be needed to determine the class association of the features and learn accurate weights for them. Al- ternatively, the user may directly label the features. For example, for a sentiment classification task, the user may label features, such as words or phrases, as expressing positive or negative sentiment. Prior work (Raghavan et al., 2006; Zaidan et al., 2007) has demonstrated that users are able to reliably pro- vide useful feedback on features. Direct feedback on a list of features (Raghavan et al., 2006; Druck et al., 2008) is limited to simple fea- tures like unigrams. However, unigrams are limited in the linguistic phenomena they can capture. Struc- tured features such as dependency relations, paths in syntactic parse trees, etc., are often needed for learn- ing the target concept (Pradhan et al., 2004; Joshi and Ros´e, 2009). It is not clear how direct feature feedback can be extended straightforwardly to struc- tured features, as they are difficult to present visu- ally for feedback and may require special expertise to comprehend. An alternative approach is to seek indirect feedback on structured features (Arora and Nyberg, 2009) by asking the user to highlight spans of text, called rationales, that support the instance label (Zaidan et al., 2007). For example, when clas- sifying the sentiment of a movie review, rationales are spans of text in the review that support the senti- ment label for the review. Assuming a fixed cost per unit of work, it might be cheaper to …

Content truncated to 500 words