Sort by Structure: Language Model Ranking as Dependency Probing

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1296 - 1307 July 10-15, 2022 ©2022 Association for Computational Linguistics � Sort by Structure: Language Model Ranking as Dependency Probing Max Müller-Eberstein ☼ and Rob van der Goot ☼ and Barbara Plank ☼� ☼ Department of Computer Science, IT University of Copenhagen, Denmark � Center for Information and Language Processing (CIS), LMU Munich, Germany mamy@itu.dk, robv@itu.dk, bplank@cis.lmu.de Abstract Making an informed choice of pre-trained lan- guage model (LM) is critical for performance, yet environmentally costly, and as such widely underexplored. The field of Computer Vi- sion has begun to tackle encoder ranking, with promising forays into Natural Language Pro- cessing, however they lack coverage of linguis- tic tasks such as structured prediction. We propose probing to rank LMs, specifically for parsing dependencies in a given language, by measuring the degree to which labeled trees are recoverable from an LM’s contextualized embeddings. Across 46 typologically and ar- chitecturally diverse LM-language pairs, our probing approach predicts the best LM choice 79% of the time using orders of magnitude less compute than training a full parser. Within this study, we identify and analyze one recently pro- posed decoupled LM—RemBERT—and find it strikingly contains less inherent dependency in- formation, but often yields the best parser after full fine-tuning. Without this outlier our ap- proach identifies the best LM in 89% of cases. 1 Introduction With the advent of massively pre-trained language models (LMs) in Natural Language Processing (NLP), it has become crucial for practitioners to choose the best LM encoder for their given task early on, regardless of the rest of their proposed model architecture. The greatest variation of LMs lies in the language or domain-specificity of the unlabelled data used during pre-training (with ar- chitectures often staying identical). Typically, better expressivity is expected from language/domain-specific LMs (Gururangan et al., 2020; Dai et al., 2020) while open-domain settings necessitate high-capacity models with access to as much pre-training data as possible. This tradeoff is difficult to navigate, and given that multiple special- ized LMs (or none at all) are available, practitioners often resort to an ad-hoc choice. In absence of im- mediate performance indicators, the most accurate choice could be made by training the full model using each LM candidate, however this is often infeasible and wasteful (Strubell et al., 2019). Recently, the field of Computer Vision (CV) has attempted to tackle this problem by quantifying useful information in pre-trained image encoders as measured directly on labeled target data without fine-tuning (Nguyen et al., 2020; You et al., 2021). While first forays for applying these methods to NLP are promising, some linguistic tasks differ substantially: Structured prediction, such as pars- ing syntactic dependencies, is a fundamental NLP task not covered by prior encoder ranking methods due to its graphical output. Simultaneously, perfor- mance prediction in NLP has so far been studied as a function of dataset and model characteristics (Xia et al., 2020; Ye …

Content truncated to 500 words