CAMERO: Consistency Regularized Ensemble of Perturbed Language Models with Weight Sharing

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7162 - 7175 May 22-27, 2022 c⃝2022 Association for Computational Linguistics CAMERO: Consistency Regularized Ensemble of Perturbed Language Models with Weight Sharing Chen Liang ∗ Georgia Institute of Technology cliang73@gatech.edu Pengcheng He, Yelong Shen Microsoft Azure AI {penhe,yelong.shen}@microsoft.com Weizhu Chen Microsoft Azure AI wzchen@microsoft.com Tuo Zhao Georgia Institute of Technology tourzhao@gatech.edu Abstract Model ensemble is a popular approach to produce a low-variance and well-generalized model. However, it induces large memory and inference costs, which are often not af- fordable for real-world deployment. Existing work has resorted to sharing weights among models. However, when increasing the propor- tion of the shared weights, the resulting mod- els tend to be similar, and the benefits of using model ensemble diminish. To retain ensemble benefits while maintaining a low memory cost, we propose a consistency-regularized ensem- ble learning approach based on perturbed mod- els, named CAMERO. Specifically, we share the weights of bottom layers across all mod- els and apply different perturbations to the hidden representations for different models, which can effectively promote the model diver- sity. Meanwhile, we apply a prediction con- sistency regularizer across the perturbed mod- els to control the variance due to the model diversity. Our experiments using large lan- guage models demonstrate that CAMERO sig- nificantly improves the generalization perfor- mance of the ensemble model. Specifically, CAMERO outperforms the standard ensemble of 8 BERT-base models on the GLUE bench- mark by 0.7 with a significantly smaller model size (114.2M vs. 880.6M). 1 Introduction Deep Neural Networks (DNNs) have achieved re- markable success in various fields and have become very powerful in learning complicated models (De- vlin et al., 2018; Brown et al., 2020; He et al., 2020). However, their remarkable representation powers come at the expense of large model variance, which may hurt the model generalization performance. A popular approach for reducing such variance is model ensemble, where the weights or predictions of a set of models are aggregated to produce the ∗Work was done during an internship at Microsoft Azure AI. predictions (Yang and Lv, 2021; Dong et al., 2020). For example, Zhang et al. (2018) show that a simple 2-model ensemble leads to notable improvement over a single model in computer vision tasks. Despite such notable benefits, model ensemble has not been widely applied to large language mod- els. The major barriers are its enormous storage and expensive inference cost, which linearly scales with the size and the number of models. Therefore, it is often not affordable to ensemble large language models for deployment using memory-constrained and low-latency edge devices. To alleviate the memory burden, recent works have resorted to a weight-sharing strategy, where all models share the same set of bottom-layer weights, on top of which branches out a set of parallel, un-shared top-layer weights (Lan et al., 2018; Chen et al., 2020; Li et al., 2020). Since the shared weights are optimized to accommodate multiple diverse un-shared branches, …

Content truncated to 500 words