Active Learning for BERT: An Empirical Study

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7949–7962, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7949 Active Learning for BERT: An Empirical Study Liat Ein-Dor∗, Alon Halfon∗, Ariel Gera∗, Eyal Shnarch∗, Lena Dankin, Leshem Choshen Marina Danilevsky, Ranit Aharonov, Yoav Katz and Noam Slonim IBM Research {liate, alonhal, arielge, eyals, lenad, leshem.choshen, katz, noams}@il.ibm.com, mdanile@us.ibm.com, ranit.aharonov2@ibm.com Abstract Real world scenarios present a challenge for text classification, since labels are usually ex- pensive and the data is often characterized by class imbalance. Active Learning (AL) is a ubiquitous paradigm to cope with data scarcity. Recently, pre-trained NLP models, and BERT in particular, are receiving massive attention due to their outstanding performance in var- ious NLP tasks. However, the use of AL with deep pre-trained models has so far re- ceived little consideration. Here, we present a large-scale empirical study on active learn- ing techniques for BERT-based classification, addressing a diverse set of AL strategies and datasets. We focus on practical scenarios of binary text classification, where the annota- tion budget is very small, and the data is of- ten skewed. Our results demonstrate that AL can boost BERT performance, especially in the most realistic scenario in which the ini- tial set of labeled examples is created using keyword-based queries, resulting in a biased sample of the minority class. We release our research framework, aiming to facilitate future research along the lines explored here. 1 Introduction Automatic text classification is a well studied prob- lem in Natural Language Processing (NLP), with great practical importance and numerous real world applications (Aggarwal and Zhai, 2012). There are two major hurdles to developing effective text clas- sifiers in practice, as well as to developing classi- fiers in other domains – the lack of labeled data, and class imbalance (Japkowicz and Stephen, 2002). Text classifiers often require high quantities of la- beled data for model training. However, collecting such labeled data is a notoriously expensive and time-consuming process, and shortage of labeled data is exacerbated when the desired class has a ∗These authors equally contributed to this work. relatively low prior in the data. In such a scenario, even going through the burden of labeling a ran- dom sample may yield an insufficient number of positive instances to properly train a classifier. Our focus in this work is on this challenging coupled setup, frequently encountered by real-world users – where labeled data is scarce and the prior of the desired class is small. A classical approach for coping with limited an- notation resources is Active Learning (AL) (Cohn et al., 1996). In this paradigm, one assumes that un- labeled data are abundant, and the goal is to focus the expensive labeling process on the most infor- mative instances. Many AL strategies have been proposed, aiming to minimize the labeling burden, or if taken from a different perspective – maximize the value of labeling a small set of examples. Im- portantly, the usefulness of an AL strategy naturally depends on …

Content truncated to 500 words