Exploring Mode Connectivity for Pre-trained Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6726 - 6746 December 7-11, 2022 ©2022 Association for Computational Linguistics Exploring Mode Connectivity for Pre-trained Language Models Yujia Qin1∗, Cheng Qian1∗, Jing Yi1∗, Weize Chen1, Yankai Lin2,3† , Xu Han1, Zhiyuan Liu1,4,5†, Maosong Sun1,4,5†, Jie Zhou6 1NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing 2Gaoling School of Artificial Intelligence, Renmin University of China, Beijing 3Beijing Key Laboratory of Big Data Management and Analysis Methods, Beijing 4International Innovation Center of Tsinghua University, Shanghai 5Quan Cheng Laboratory 6Pattern Recognition Center, WeChat AI, Tencent Inc. {qyj20, qianc20, yi-j20}@mails.tsinghua.edu.cn Abstract Recent years have witnessed the prevalent application of pre-trained language models (PLMs) in NLP. From the perspective of pa- rameter space, PLMs provide generic initial- ization, starting from which high-performance minima could be found. Although plenty of works have studied how to effectively and effi- ciently adapt PLMs to high-performance min- ima, little is known about the connection of various minima reached under different adap- tation configurations. In this paper, we in- vestigate the geometric connections of differ- ent minima through the lens of mode con- nectivity, which measures whether two min- ima can be connected with a low-loss path. We conduct empirical analyses to investigate three questions: (1) how could hyperparame- ters, specific tuning methods, and training data affect PLM’s mode connectivity? (2) How does mode connectivity change during pre- training? (3) How does the PLM’s task knowl- edge change along the path connecting two minima? In general, exploring the mode con- nectivity of PLMs conduces to understanding the geometric connection of different minima, which may help us fathom the inner workings of PLM downstream adaptation. The codes are publicly available at https://github.com/ thunlp/Mode-Connectivity-PLM. 1 Introduction Recent years have witnessed the prevalent appli- cation of pre-trained language models (PLMs) in NLP (Han et al., 2021), with the state-of-the- art across various NLP tasks consistently being pushed (Devlin et al., 2019; Liu et al., 2019b; Raffel et al., 2020). Through large-scale self-supervised training, PLMs acquire versatile semantic (Liu ∗Indicates equal contribution. †Corresponding author. et al., 2019a) and syntactic (Tenney et al., 2019) knowledge, which could be utilized when conduct- ing transfer learning on downstream tasks. From the perspective of parameter space, PLMs provide generic initialization for downstream adap- tation. Starting from the initialization, many high-performance minima can be found through gradient-based optimization. Up to now, plenty of works have studied how to effectively and ef- ficiently adapt PLMs to high-performance min- ima, including adjusting hyperparameters (Liu and Wang, 2021), conducting transfer learning using auxiliary training data (Pruksachatkun et al., 2020), tuning PLMs in a parameter-efficient way (Ding et al., 2022), etc. Under different adaptation con- figurations, PLMs may finally reach local minima distributed in highly distinct regions. Although these minima all correspond to excellent perfor- mance (low loss), little has been known about their geometric connection in the parameter space. A straightforward way to explore such geomet- ric connection is to look into the loss landscape around different minima, which …

Content truncated to 500 words