BERT Prescriptions to Avoid Unwanted Headaches: A Comparison of Transformer Architectures for Adverse Drug Event Detection

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 187–199 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 187 Maximal Multiverse Learning for Promoting Cross-Task Generalization of Fine-Tuned Language Models Itzik Malkiel Tel Aviv University itzik.malkiel@gmail.com Lior Wolf Tel Aviv University wolf@cs.tau.ac.il Abstract Language modeling with BERT consists of two phases of (i) unsupervised pre-training on unlabeled text, and (ii) fine-tuning for a specific supervised task. We present a method that leverages the second phase to its fullest, by applying an extensive num- ber of parallel classifier heads, which are enforced to be orthogonal, while adaptively eliminating the weaker heads during train- ing. We conduct an extensive inter- and intra- dataset evaluation, showing that our method improves the generalization ability of BERT, sometimes leading to a +9% gain in accuracy. These results highlight the importance of a proper fine-tuning procedure, especially for relatively smaller-sized datasets. Our code is attached as supplementary. 1 Introduction Recently, there has been an increasing number of studies suggesting the use of general language models, for improving natural language process- ing tasks (Dai and Le, 2015; Peters et al., 2018; Radford et al., 2018; Howard and Ruder, 2018). Among the most promising techniques, the unsu- pervised pretraining approach (Dai and Le, 2015; Radford et al., 2018) has emerged as a very suc- cessful method, that achieves state-of-the-art re- sults on many language tasks, including senti- ment analysis (Socher et al., 2013), natural lan- guage inference (Williams et al., 2017) and similar- ity and paraphrasing tasks (Dolan and Brockett, 2005; Cer et al., 2017). This approach incorpo- rates a two-phase training procedure. The first phase utilizes an unsupervised training of a gen- eral language model on a large corpus. The sec- ond phase applies supervision to fine-tune the model for a given task. More recently, unsupervised pretraining mod- els such as BERT (Devlin et al., 2018), XLNET (Yang et al., 2019) and RoBERTa (Liu et al., 2019), have achieved unprecedented performance. For example, in the GLUE benchmark (Wang et al., 2018), BERT (Devlin et al., 2018) was reported to achieve performance that exceeds human level on a few different datasets, such as QNLI (Ra- jpurkar et al., 2016), QQP (Chen et al., 2018) and MRPC (Dolan and Brockett, 2005). However, de- spite the great progress achieved by these task- specific and dataset-specific models, it is not yet clear how well they can generalize to different tasks, how well they generalize when evaluating the same task on different datasets, and how to improve this generalization ability. In our work, we extend the multiverse method of (Littwin and Wolf, 2016), which was shown to improve transfer learning in the computer vision task of face recognition and on the CIFAR-100 small image recognition dataset. The multiverse loss generalizes the cross entropy loss, by simul- taneously training multiple linear classification heads to perform the same task. In order to pre- vent multiple copies of the same classifier, in the multiverse scheme, each …

Content truncated to 500 words