Evidence > Intuition: Transferability Estimation for Encoder Selection

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4218 - 4227 December 7-11, 2022 ©2022 Association for Computational Linguistics Evidence > Intuition: Transferability Estimation for Encoder Selection Elisa Bassignana � ☼ Max Müller-Eberstein � ☼ Mike Zhang � ☼ Barbara Plank☼�� ☼Department of Computer Science, IT University of Copenhagen, Denmark �Center for Information and Language Processing (CIS), LMU Munich, Germany �Munich Center for Machine Learning (MCML), Munich, Germany {elba, mamy, mikz}@itu.dk b.plank@lmu.de Abstract With the increase in availability of large pre-trained language models (LMs) in Natural Language Processing (NLP), it becomes critical to assess their fit for a specific target task a priori—as fine-tuning the entire space of available LMs is computationally prohibitive and unsustainable. However, encoder trans- ferability estimation has received little to no attention in NLP. In this paper, we propose to generate quantitative evidence to predict which LM, out of a pool of models, will perform best on a target task without having to fine-tune all candidates. We provide a comprehensive study on LM ranking for 10 NLP tasks spanning the two fundamental problem types of classi- fication and structured prediction. We adopt the state-of-the-art Logarithm of Maximum Evidence (LogME) measure from Computer Vision (CV) and find that it positively corre- lates with final LM performance in 94% of the setups. In the first study of its kind, we further compare transferability measures with the de facto standard of human practitioner ranking, finding that evidence from quantitative metrics is more robust than pure intuition and can help identify unexpected LM candidates. 1 Introduction Advances in Deep Learning-based NLP and CV build on expressive representations from encoder models pre-trained on massive corpora. Down- stream models make use of latent information in these representations to extract relevant features for the task at hand. Within this paradigm, decid- ing which pre-trained encoder to use in any task- specific architecture is crucial, however training a model using each encoder candidate is infeasi- ble. In absence of prior heuristics (e.g., via related work), the choice of encoder has therefore prevail- ingly been based on practitioner intuition rather than quantitative evidence. � The authors contributed equally to this work. In NLP, prior work has examined the different yet related task of performance prediction (Xia et al., 2020a; Ye et al., 2021), surveyed and cat- egorized LMs (Xia et al., 2020b), and used probing to predict LM performance specifically for depen- dency parsing (Müller-Eberstein et al., 2022b), but has yet to extensively investigate how to rank the increasingly large number of pre-trained LM en- coders across various tasks and domains. Prelim- inary work by You et al. (2021) shows that the LogME estimator holds promise, including the first steps for encoder selection in NLP. With their main focus being on CV, however, they evaluate only a limited set of tasks and models for NLP and use self-reported benchmark scores instead of running controlled experiments which should include, e.g., the variance across initializations, domains, and fine-tuning strategies (Section 2). As …

Content truncated to 500 words