Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1698–1712 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1698 Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates Artem Shelmanov1, Dmitri Puzyrev1,2,3, Lyubov Kupriyanova1, Denis Belyakov2, Daniil Larionov2,3,4, Nikita Khromov1, Olga Kozlova3, Ekaterina Artemova2, Dmitry V. Dylov1, Alexander Panchenko1 1Skolkovo Institute of Science and Technology, Russia 2Higher School of Economics University, Russia 3Mobile TeleSystems (MTS), Russia 4Federal Research Center “Computer Science and Control” of Russian Academy of Sciences {a.shelmanov,dmitrii.puzyrev,lyubov.kupriyanova, nikita.khromov,d.dylov,a.panchenko}@skoltech.ru {dobelyakov,dslarionov}@edu.hse.ru, elartemova@hse.ru oskozlo9@mts.ru Abstract Annotating training data for sequence tagging of texts is usually very time-consuming. Re- cent advances in transfer learning for natural language processing in conjunction with active learning open the possibility to significantly re- duce the necessary annotation budget. We are the first to thoroughly investigate this powerful combination for the sequence tagging task. We conduct an extensive empirical study of vari- ous Bayesian uncertainty estimation methods and Monte Carlo dropout options for deep pre- trained models in the active learning frame- work and find the best combinations for differ- ent types of models. Besides, we also demon- strate that to acquire instances during active learning, a full-size Transformer can be sub- stituted with a distilled version, which yields better computational performance and reduces obstacles for applying deep active learning in practice. 1 Introduction In many natural language processing (NLP) tasks, such as named entity recognition (NER), obtaining gold standard labels for constructing the training dataset can be very time and labor-consuming. It makes the annotation process expensive and lim- its the application of supervised machine learning models. This is especially the case in such domains as biomedical or scientific text processing, where crowdsourcing is either difficult or prohibitively expensive. In these domains, highly-qualified ex- perts are needed to annotate data correctly, which dramatically increases the annotation cost. Active Learning (AL) is a technique that can help to reduce the amount of annotation required to train a good model by multiple times (Settles and Craven, 2008; Settles, 2009). Opposite to ex- haustive and redundant manual annotation of the entire corpus, AL drives the annotation process to focus the expensive human expert time only on the most informative objects, which contributes to a substantial increase in the model quality. AL is an iterative process that starts from a small number of labeled seeding instances. In each iter- ation, an acquisition model is trained on the cur- rently annotated dataset and is applied to the large pool of unannotated objects. The model predic- tions are used by the AL query strategy to sam- ple the informative objects, which are then further demonstrated to the expert annotators. When the annotators provide labels for these objects, the next iteration begins. The collected data can be used for training a final successor model that is used in a target application. During AL, acquisition models have to be trained on very small amounts of the labeled data, especially during the …

Content truncated to 500 words