BAM! Born-Again Multi-Task Networks for Natural Language Understanding

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5931–5937 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5931 BAM! Born-Again Multi-Task Networks for Natural Language Understanding Kevin Clark† Minh-Thang Luong‡ Urvashi Khandelwal† Christopher D. Manning† Quoc V. Le‡ †Computer Science Department, Stanford University ‡ Google Brain {kevclark,urvashik,manning}@cs.stanford.edu {thangluong,qvl}@google.com Abstract It can be challenging to train multi-task neu- ral networks that outperform or even match their single-task counterparts. To help address this, we propose using knowledge distillation where single-task models teach a multi-task model. We enhance this training with teacher annealing, a novel method that gradually tran- sitions the model from distillation to super- vised learning, helping the multi-task model surpass its single-task teachers. We evaluate our approach by multi-task fine-tuning BERT on the GLUE benchmark. Our method consis- tently improves over standard single-task and multi-task training. 1 Introduction Building a single model that jointly learns to per- form many tasks effectively has been a long- standing challenge in Natural Language Process- ing (NLP). However, multi-task NLP remains dif- ficult for many applications, with multi-task mod- els often performing worse than their single-task counterparts (Plank and Alonso, 2017; Bingel and Søgaard, 2017; McCann et al., 2018). Motivated by these results, we propose a way of applying knowledge distillation (Bucilu et al., 2006; Ba and Caruana, 2014; Hinton et al., 2015) so that single- task models effectively teach a multi-task model. Knowledge distillation transfers knowledge from a “teacher” model to a “student” model by training the student to imitate the teacher’s out- puts. In “born-again networks” (Furlanello et al., 2018), the teacher and student have the same neu- ral architecture and model size, but surprisingly the student is able to surpass the teacher’s accu- racy. Intuitively, distillation is effective because the teacher’s output distribution over classes pro- vides more training signal than a one-hot label; Hinton et al. (2015) suggest that teacher outputs contain “dark knowledge” capturing additional in- formation about training examples. 0 time train distill train Task 1 Model Task 2 Model Task k Model Multi-Task Model Task 1 Labels Task 2 Labels Task k Labels 1 Figure 1: Overview of our method. λ is increased lin- early from 0 to 1 over the course of training. Our work extends born-again networks to the multi-task setting. We compare Single→Multi1 born-again distillation with several other variants (Single→Single and Multi→Multi), and also ex- plore performing multiple rounds of distillation (Single→Multi→Single→Multi). Furthermore, we propose a simple teacher annealing method that helps the student model outperform its teach- ers. Teacher annealing gradually transitions the student from learning from the teacher to learn- ing from the gold labels. This method ensures the student gets a rich training signal early in training, but is not limited to only imitating the teacher. Our experiments build upon recent success in self-supervised pre-training (Dai and Le, 2015; Peters et al., 2018) and multi-task fine-tune BERT (Devlin et al., 2019) to perform the tasks from the GLUE natural language understanding …

Content truncated to 500 words