Active Learning with Rationales for Text Classification

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 441–451, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Active Learning with Rationales for Text Classification Manali Sharma, Di Zhuang and Mustafa Bilgic Department of Computer Science Illinois Institute of Technology Chicago, IL USA {msharm11, dzhuang3}@hawk.iit.edu and mbilgic@iit.edu Abstract We present a simple and yet effective ap- proach that can incorporate rationales elicited from annotators into the training of any off- the-shelf classifier. We show that our sim- ple approach is effective for multinomial na¨ıve Bayes, logistic regression, and support vector machines. We additionally present an active learning method tailored specifically for the learning with rationales framework. 1 Introduction Annotating documents for supervised learning is a tedious, laborious, and time consuming task for hu- mans. Given huge amounts of unlabeled documents, it is impractical for annotators to go over each doc- ument and provide a label. To reduce the anno- tation time and effort, various approaches such as semi-supervised learning (Chapelle et al., 2006) that utilizes both labeled and unlabeled data, and active learning (Settles, 2012) that carefully chooses in- stances for annotation have been developed. To fur- ther minimize the human effort, recent work looked at eliciting domain knowledge, such as rationales and feature annotations, from the annotators instead of just the labels of documents. One of the bottlenecks in eliciting domain knowl- edge from annotators is that the traditional super- vised learning approaches cannot readily handle the elicited rich feedback. To address this issue, many methods have been developed that are classifier- specific. Examples include knowledge-based neural networks (e.g., (Towell and Shavlik, 1994), (Girosi and Chan, 1995), (Towell et al., 1990)), knowledge- based support vector machines (Fung et al., 2002), pooling multinomial na¨ıve Bayes (Melville and Sindhwani, 2009), incorporating constraints into the training of na¨ıve Bayes (Stumpf et al., 2007), and converting rationales and feature annotations into constraints for support vector machines (e.g., (Small et al., 2011) and (Zaidan et al., 2007)). Being classifier-specific limits their applicability when one wants to test a different classifier for his/her domain, necessitating an approach that can be utilized by sev- eral off-the-shelf classifiers. In this paper we present a simple and yet effective approach that can incorporate the elicited rationales in the form of feature annotations into the training of any off-the-shelf classifier. We empirically show that it is effective at incorporating rationales into the learning of na¨ıve Bayes, logistic regression, and sup- port vector machines using four text categorization datasets. We further discuss a novel active learn- ing strategy specifically geared towards the learning with rationales framework and empirically show that it improves over traditional active learning. The rest of the paper is organized as follows. In Section 2, we provide a brief background on elicit- ing rationales in the context of active learning. In Section 3, we describe our approach for incorpo- rating rationales into the training of classifiers and compare learning without rationales and learning with rationales. …

Content truncated to 500 words