To BERT or Not to BERT: Comparing Task-specific and Task-agnostic Semi-Supervised Approaches for Sequence Tagging

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7927–7934, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7927 To BERT or Not to BERT: Comparing Task-specific and Task-agnostic Semi-Supervised Approaches for Sequence Tagging Kasturi Bhattacharjee Miguel Ballesteros Rishita Anubhai Smaranda Muresan∗ Jie Ma Faisal Ladhak Yaser Al-Onaizan Amazon AI {kastb, ballemig, ranubhai, smaranm, jieman, faisall, onaizan}@amazon.com Abstract Leveraging large amounts of unlabeled data us- ing Transformer-like architectures, like BERT, has gained popularity in recent times owing to their effectiveness in learning general rep- resentations that can then be further fine-tuned for downstream tasks to much success. How- ever, training these models can be costly both from an economic and environmental stand- point. In this work, we investigate how to ef- fectively use unlabeled data: by exploring the task-specific semi-supervised approach, Cross- View Training (CVT) and comparing it with task-agnostic BERT in multiple settings that include domain and task relevant English data. CVT uses a much lighter model architecture and we show that it achieves similar perfor- mance to BERT on a set of sequence tagging tasks, with lesser financial and environmental impact. 1 Introduction Exploiting unlabeled data to improve performance has become the foundation for many natural lan- guage processing tasks. The question we inves- tigate in this paper is how to effectively use un- labeled data: in a task-agnostic or a task-specific way. An example of the former is training models on language model (LM) like objectives on a large unlabeled corpus to learn general representations, as in ELMo (Embeddings from Language Models) (Peters et al., 2018) and BERT (Bidirectional En- coder Representations from Transformers) (Devlin et al., 2019). These are then reused in supervised training on a downstream task. These pre-trained models, particularly the ones based on the Trans- former architecture (Vaswani et al., 2017)1 have ∗ Smaranda Muresan is an Amazon Scholar and a Re- search Scientist at the Data Science Institute, Columbia Uni- versity 1Not only BERT, but other models like RoBERTa (Liu et al., 2019b) and BART (Lewis et al., 2019) achieved state-of-the-art results in a variety of NLP tasks, but come at a great cost financially and envi- ronmentally (Strubell et al., 2019; Schwartz et al., 2019). In contrast, Cross-View Training (CVT) (Clark et al., 2018) is a semi-supervised approach that uses unlabeled data in a task-specific manner, rather than trying to learn general representations that can be used for many downstream tasks. Inspired by self- learning (McClosky et al., 2006; Yarowsky, 1995) and multi-view learning (Blum and Mitchell, 1998; Xu et al., 2013), the key idea is that the primary pre- diction module, which has an unrestricted view of the data, trains on the task using labeled examples, and makes task-specific predictions on unlabeled data. The auxiliary modules, with restricted views of the unlabeled data, attempt to replicate the pri- mary module predictions. This helps to learn better representations for the task. We present an experimental study that investi- gates different task-agnostic and task-specific ap- proaches to use …

Content truncated to 500 words