Practical Obstacles to Deploying Active Learning

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 21–30, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 21 Practical Obstacles to Deploying Active Learning David Lowell Northeastern University lowell.d@husky.neu.edu Zachary C. Lipton Carnegie Mellon University zlipton@cmu.edu Byron C. Wallace Northeastern University b.wallace@northeastern.edu Abstract Active learning (AL) is a widely-used train- ing strategy for maximizing predictive perfor- mance subject to a fixed annotation budget. In AL one iteratively selects training examples for annotation, often those for which the cur- rent model is most uncertain (by some mea- sure). The hope is that active sampling leads to better performance than would be achieved under independent and identically distributed (i.i.d.) random samples. While AL has shown promise in retrospective evaluations, these studies often ignore practical obstacles to its use. In this paper we show that while AL may provide benefits when used with spe- cific models and for particular domains, the benefits of current approaches do not general- ize reliably across models and tasks. This is problematic because in practice one does not have the opportunity to explore and compare alternative AL strategies. Moreover, AL cou- ples the training dataset with the model used to guide its acquisition. We find that sub- sequently training a successor model with an actively-acquired dataset does not consistently outperform training on i.i.d. sampled data. Our findings raise the question of whether the downsides inherent to AL are worth the mod- est and inconsistent performance gains it tends to afford. 1 Introduction Although deep learning now achieves state-of- the-art results on a number of supervised learn- ing tasks (Johnson and Zhang, 2016; Ghaddar and Langlais, 2018), realizing these gains requires large annotated datasets (Shen et al., 2018). This data dependence is problematic because labels are expensive. Several lines of research seek to reduce the amount of supervision required to achieve ac- ceptable predictive performance, including semi- supervised (Chapelle et al., 2009), transfer (Pan and Yang, 2010), and active learning (AL) (Cohn et al., 1996; Settles, 2012). In AL, rather than training on a set of labeled data sampled at i.i.d. random from some larger population, the learner engages the annotator in a cycle of learning, iteratively selecting training data for annotation and updating its model. Pool- based AL (the variant we consider) proceeds in rounds. In each, the learner applies a heuristic to score unlabeled instances, selecting the highest scoring instances for annotation.1 Intuitively, by selecting training data cleverly, an active learner might achieve greater predictive performance than it would by choosing examples at random. The more informative samples come at the cost of violating the standard i.i.d. assumption upon which supervised machine learning typically re- lies. In other words, the training and test data no longer reflect the same underlying data distribu- tion. Empirically, AL has been found to work well with a variety of tasks and models (Settles, 2012; Ramirez-Loaiza et al., 2017; Gal et al., 2017a; Zhang …

Content truncated to 500 words