Multi-task Active Learning for Pre-trained Transformer-based Models

Multi-task Active Learning for Pre-trained Transformer-based Models Guy Rotman and Roi Reichart Faculty of Industrial Engineering and Management, Technion, IIT, Israel grotman@campus.technion.ac.il roiri@technion.ac.il Abstract Multi-task learning, in which several tasks are jointly learned by a single model, allows NLP models to share information from multi- ple annotations and may facilitate better pre- dictions when the tasks are inter-related. This technique, however, requires annotating the same text with multiple annotation schemes, which may be costly and laborious. Active learning (AL) has been demonstrated to op- timize annotation processes by iteratively se- lecting unlabeled examples whose annotation is most valuable for the NLP model. Yet, multi-task active learning (MT-AL) has not been applied to state-of-the-art pre-trained Transformer-based NLP models. This paper aims to close this gap. We explore various multi-task selection criteria in three realistic multi-task scenarios, reflecting different re- lations between the participating tasks, and demonstrate the effectiveness of multi-task compared to single-task selection. Our results suggest that MT-AL can be effectively used in order to minimize annotation efforts for multi-task NLP models.1 1 Introduction Deep neural networks (DNNs) have recently achieved state-of-the-art results for many natural language processing (NLP) tasks and applica- tions. Of particular importance are contextual- ized embedding models (McCann et al., 2017; Peters et al., 2018), most of which imple- ment Transformer-based architectures with the self-attention mechanism (Vaswani et al., 2017; Devlin et al., 2019; Raffel et al., 2020). Nevertheless, DNNs often require large labeled training sets in order to achieve good performance. While annotating such training sets is costly and laborious, the active learning (AL) paradigm aims to minimize these costs by iteratively selecting 1Our code base is available at: https://github.com /rotmanguy/MTAL. valuable training examples for annotation. Re- cently, AL has been shown effective for DNNs across various NLP tasks (Duong et al., 2018; Peris and Casacuberta, 2018; Ein-Dor et al., 2020). An appealing capability of DNNs is perform- ing multi-task learning (MTL): Learning multiple tasks by a single model (Ruder, 2017). This stems from their architectural flexibility—constructing increasingly deeper and wider architectures from basic building blocks—and in their gradient-based optimization, which allows them to jointly update parameters from multiple task-based objectives. Indeed, MTL has become ubiquitous in NLP (Luan et al., 2018; Liu et al., 2019a). MTL models for NLP can often benefit from using corpora annotated for multiple tasks, par- ticularly when these tasks are closely related and can inform each other. Prominent examples of multi-task corpora include OntoNotes (Hovy et al., 2006), the Universal Dependencies Bank (Nivre et al., 2020), and STREUSLE (Schneider et al., 2018). Given the importance of multi-task corpora for many MTL setups, effective AL frameworks that support MTL are becoming crucial. Unfortunately, most AL methods do not support annotations for more than one task. Multi-task AL (MT-AL) was proposed by Reichart et al. (2008) before the neural era, and adapted by Ikhwantri et al. (2018) to a neural architecture. Recently, Zhu et al. (2020) proposed an MT-AL model for slot filling and intent detection, focusing mostly on LSTMs (Hochreiter …

Content truncated to 500 words