An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2286–2300 August 1–6, 2021. ©2021 Association for Computational Linguistics 2286 An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models Xueqing Liu Stevens Institute of Technology xueqing.liu@stevens.edu Chi Wang Microsoft Research wang.chi@microsoft.com Abstract The performance of fine-tuning pre-trained language models largely depends on the hyper- parameter configuration. In this paper, we in- vestigate the performance of modern hyperpa- rameter optimization methods (HPO) on fine- tuning pre-trained language models. First, we study and report three HPO algorithms’ per- formances on fine-tuning two state-of-the-art language models on the GLUE dataset. We find that using the same time budget, HPO of- ten fails to outperform grid search due to two reasons: insufficient time budget and overfit- ting. We propose two general strategies and an experimental procedure to systematically trou- bleshoot HPO’s failure cases. By applying the procedure, we observe that HPO can succeed with more appropriate settings in the search space and time budget; however, in certain cases overfitting remains. Finally, we make suggestions for future work. Our implemen- tation can be found in https://github.c om/microsoft/FLAML/tree/main/flaml /nlp/. 1 Introduction In the recent years, deep learning and pre-trained language models (Devlin et al., 2019; Liu et al., 2019; Clark et al., 2020; He et al., 2021) have achieved great success in the NLP community. It has now become a common practice for researchers and practitioners to fine-tune pre-trained language models in down-stream NLP tasks. For example, the HuggingFace transformers library (Wolf et al., 2020) was ranked No.1 among the most starred NLP libraries on GitHub using Python1. Same as other deep learning models, the perfor- mance of fine-tuning pre-trained language mod- els largely depends on the hyperparameter con- figuration. A different setting in the hyperparam- 1https://github.com/EvanLi/Github- Ranking/blob/master/Top100/Python.md eters may cause a significant drop in the perfor- mance, turning a state-of-the-art model into a poor model. Methods for tuning hyperparameters can be categorized as (1) traditional approaches such as manual tuning and grid search, and (2) automated HPO methods such as random search and Bayesian optimization (BO). Manual tuning often requires a large amount of manual efforts; whereas grid search often suffers from lower efficiency due to the exponential increase in time cost with the num- ber of hyperparameters. Automated HPO methods were proposed to overcome these disadvantages. Recently, automated HPO methods also become in- creasingly popular in the NLP community (Zhang and Duh, 2020; Dodge et al., 2019). For exam- ple, Bayesian optimization (BO) (Zhang and Duh, 2020) and Population-based Training (Jaderberg et al., 2017) both prove to be helpful for improving the performance of the transformer model (Vaswani et al., 2017) for neural machine translation. The HuggingFace library has also added native sup- ports for HPO in a recent update (version 3.1.0, Aug 2020). With improved supports, users can now easily access a variety of HPO methods and apply them to their fine-tuning tasks. However, the effectiveness of this …

Content truncated to 500 words