Multiple Teacher Distillation for Robust and Greener Models

Proceedings of Recent Advances in Natural Language Processing, pages 601–610 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_068 601 Multiple Teacher Distillation for Robust and Greener Models Artur Ilichev0 Moscow Institute of Physics and Technology, Moscow, Russia ilichev.as@phystech.edu Nikita Sorokin Valentin Malykh Huawei Noah’s Ark lab, Moscow, Russia lastname.firstname@huawei.com Irina Piontkovskaya Abstract The language models nowadays are in the cen- ter of natural language processing progress. These models are mostly of significant size. There are successful attempts to reduce them, but at least some of these attempts rely on ran- domness. We propose a novel distillation pro- cedure leveraging on multiple teachers usage which alleviates random seed dependency and makes the models more robust. We show that this procedure applied to TinyBERT and Dis- tilBERT models improves their worst case re- sults up to 2% while keeping almost the same best-case ones. The latter fact keeps true with a constraint on computational time, which is important to lessen the carbon footprint. In ad- dition, we present the results of an application of the proposed procedure to a computer vision model ResNet, which shows that the statement keeps true in this totally different domain. 1 Introduction Nowadays the language models became a corner- stone in many natural language processing tasks. Their results in the benchmarks show new high scores. But with great power sometimes comes huge size, the current models could have dozens of billions of weights, e.g. TuringNLG (Rasley et al., 2020), to GPT-3 (Brown et al., 2020) with 175 bil- lion parameters, and counting. In many cases the resources of computational memory are limited and there is a demand for small solutions. One of such solutions is a distillation of language models. There were presented several approaches for the specified task, among others these are TinyBERT (Jiao et al., 2019) and DistilBERT (Sanh et al., 2019). We analyzed these approaches and found that they share an important flaw - the dependency from the random seed used in the distillation process. 0Work done while Artur Ilichev was at Huawei. During the distillation a student model needs to be trained multiple times with different random seeds to achieve better performance, although it is not guaranteed that there will be “winning numbers” in your seed choice. So we concentrated on the worst case scenario and proposed a technique to im- prove it. Considering the computational resources, the improvement could be achieved with the same computational budget, allowing one to diminish the carbon footprint. We propose the novel tech- nique of multi-teacher distillation called to make the mentioned language models more robust to seed selection. We evaluated our method on a computer vision classification model ResNet (He et al., 2016) and make sure that the proposed technique is appli- cable to a totally different domain. Our contribution is as follows: we present (i) a new distillation method and an experimental evalu- ation of this method for three models, namely (ii) TinyBERT and (iii) DistilBERT, where we mod- ified the distillation procedure adding the task- specific distillation, for …

Content truncated to 500 words