Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learning

Proceedings of the 28th International Conference on Computational Linguistics, pages 1158–1171 Barcelona, Spain (Online), December 8-13, 2020 1158 Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learning Daniel Grießhaber Institute for Applied Artificial Intelligence (IAAI) Hochschule der Medien Stuttgart Nobelstraße 10 70569 Stuttgart griesshaber@hdm-stuttgart.de Johannes Maucher IAAI Hochschule der Medien Stuttgart Nobelstraße 10 70569 Stuttgart maucher@hdm-stuttgart.de Ngoc Thang Vu Institute for Natural Language Processing (IMS) University of Stuttgart Pfaffenwaldring 5B 70569 Stuttgart thangvu@ims.uni-stuttgart.de Abstract Recently, leveraging pre-trained Transformer based language models in down stream, task spe- cific models has advanced state of the art results in natural language understanding tasks. How- ever, only a little research has explored the suitability of this approach in low resource settings with less than 1,000 training data points. In this work, we explore fine-tuning methods of BERT - a pre-trained Transformer based language model - by utilizing pool-based active learning to speed up training while keeping the cost of labeling new data constant. Our experimental results on the GLUE data set show an advantage in model performance by maximizing the approxi- mate knowledge gain of the model when querying from the pool of unlabeled data. Finally, we demonstrate and analyze the benefits of freezing layers of the language model during fine-tuning to reduce the number of trainable parameters, making it more suitable for low-resource settings. 1 Introduction Pre-trained language models have received great interest in the natural language processing (NLP) com- munity in the last recent years (Dai and Le, 2015; Radford, 2018; Howard and Ruder, 2018; Baevski et al., 2019; Dong et al., 2019). These models are trained in a semi-supervised fashion to learn a general language model, for example, by predicting the next word of a sentence (Radford, 2018). Then, transfer learning (Pan and Yang, 2010; Ruder et al., 2019; Moeed et al., 2020) can be used to leverage the learned knowledge for a down-stream task, such as text-classification (Do and Ng, 2006; Aggarwal and Zhai, 2012; Reimers et al., 2019; Sun et al., 2019b). Devlin et al. (2019) introduced the “Bidirectional Encoder Representations from Transformers” (BERT), a pre-trained language model based on the Transformer architecture (Vaswani et al., 2017). BERT is a deeply bidirectional model that was pre-trained using a huge amount of text with a masked language model objective where the goal is to predict randomly masked words from their context (Taylor, 1953). The fact is, BERT has achieved state of the art results on the “General Language Understanding Evaluation” (GLUE) benchmark (Wang et al., 2018) by only training a single, task-specific layer at the output and fine-tuning the base model for each task. Furthermore, BERT demonstrated its applicability to many other natural language tasks since then including but not limited to sentiment analysis (Sun et al., 2019a; Xu et al., 2019; Li et al., 2019), relation extraction (Baldini Soares et al., 2019; Huang et al., 2019b) and word sense disambiguation (Huang et al., 2019a; Hadiwinoto et al., 2019; Huang et al., 2019a), as well as its adaptability to …

Content truncated to 500 words