Sampling Bias in Deep Active Classification: An Empirical Study

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4058–4068, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4058 Sampling Bias in Deep Active Classification: An Empirical Study Ameya Prabhu1∗†, Charles Dognin2∗, Maneesh Singh2 University of Oxford1 Verisk | AI, Verisk Analytics2 ameya.prabhu@mailfence.com, {charles.dognin, maneesh.singh}@verisk.com Abstract The exploding cost and time needed for data labeling and model training are bottlenecks for training DNN models on large datasets. Identi- fying smaller representative data samples with strategies like active learning can help mit- igate such bottlenecks. Previous works on active learning in NLP identify the problem of sampling bias in the samples acquired by uncertainty-based querying and develop costly approaches to address it. Using a large em- pirical study, we demonstrate that active set selection using the posterior entropy of deep models like FastText.zip (FTZ) is robust to sampling biases and to various algorithmic choices (query size and strategies) unlike that suggested by traditional literature. We also show that FTZ based query strategy pro- duces sample sets similar to those from more sophisticated approaches (e.g ensemble net- works). Finally, we show the effectiveness of the selected samples by creating tiny high- quality datasets, and utilizing them for fast and cheap training of large models. Based on the above, we propose a simple baseline for deep active text classification that outper- forms the state-of-the-art. We expect the pre- sented work to be useful and informative for dataset compression and for problems involv- ing active, semi-supervised or online learn- ing scenarios. Code and models are available at: https://github.com/drimpossible/Sampling- Bias-Active-Learning 1 Introduction Deep neural networks (DNNs) trained on large datasets provide state-of-the-art results on various NLP problems (Devlin et al., 2019) including text classification (Howard and Ruder, 2018). How- ever, the cost and time needed to get labeled data and to train models is a serious impediment to creating new and/or better models. This problem can be mitigated by creating smaller representative datasets with active learning which can be used for training DNNs to achieve similar test accuracy as ∗indicates equal contribution † Work done at Verisk | AI that using the full training dataset . In other words, the smaller sample can be considered a surrogate for the full data. However, there is lack of clarity in the active learning literature regarding sampling bias in such surrogate datasets created using active learning (Settles, 2009): its dependence on models, func- tions and parameters used to acquire the sample. Indeed, what constitutes a good sample? In this paper, we perform an empirical investigation us- ing active text classification as the application. Early work in active text classification (Lewis and Gale, 1994) suggests that greedy query gen- eration using label uncertainty may lead to ef- ficient representative samples (Nonetheless, the same test accuracy). Subsequent concerns regard- ing sampling bias has lead to explicit use of expen- sive diversity measures (Brinker, 2003; Hoi et al., 2006) in acquisition functions or using …

Content truncated to 500 words