LexFit: Lexical Fine-Tuning of Pretrained Language Models

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5269–5283 August 1–6, 2021. ©2021 Association for Computational Linguistics 5269 LEXFIT: Lexical Fine-Tuning of Pretrained Language Models Ivan Vuli´c♠ Edoardo M. Ponti♥,♠ Anna Korhonen♠ Goran Glavaš♦ ♠Language Technology Lab, University of Cambridge, UK ♥Mila - Quebec AI Institute and McGill University, Canada ♦Data and Web Science Group, University of Mannheim, Germany {iv250,alk23}@cam.ac.uk edoardo-maria.ponti@mila.quebec goran@informatik.uni-mannheim.de Abstract Transformer-based language models (LMs) pretrained on large text collections implicitly store a wealth of lexical semantic knowledge, but it is non-trivial to extract that knowledge effectively from their parameters. Inspired by prior work on semantic specialization of static word embedding (WE) models, we show that it is possible to expose and enrich lexical knowl- edge from the LMs, that is, to specialize them to serve as effective and universal “decontex- tualized” word encoders even when fed input words “in isolation” (i.e., without any context). Their transformation into such word encoders is achieved through a simple and efficient lex- ical fine-tuning procedure (termed LEXFIT) based on dual-encoder network structures. Fur- ther, we show that LEXFIT can yield effective word encoders even with limited lexical super- vision and, via cross-lingual transfer, in dif- ferent languages without any readily available external knowledge. Our evaluation over four established, structurally different lexical-level tasks in 8 languages indicates the superior- ity of LEXFIT-based WEs over standard static WEs (e.g., fastText) and WEs from vanilla LMs. Other extensive experiments and abla- tion studies further profile the LEXFIT frame- work, and indicate best practices and perfor- mance variations across LEXFIT variants, lan- guages, and lexical tasks, also directly ques- tioning the usefulness of traditional WE mod- els in the era of large neural models. 1 Introduction Probing large pretrained encoders like BERT (De- vlin et al., 2019) revealed that they contain a wealth of lexical knowledge (Ethayarajh, 2019; Vuli´c et al., 2020). If type-level word vectors are extracted from BERT with appropriate strategies, they can even outperform traditional word embeddings (WEs) in some lexical tasks (Vuli´c et al., 2020; Bommasani et al., 2020; Chronis and Erk, 2020). However, LexFit loss (w, v) = (dormant, asleep) 1. SOFTMAX 2. MNEG 3. MSIM BERT BERT Pooling Pooling Word embedding extraction w v u = sleeping BERT Step 1: Lexical fine-tuning u Step 2: Extracting word vectors from (LexFit-ed) BERT Figure 1: Illustration of the full pipeline for obtaining decontextualized word representations, based on lexi- cally fine-tuning pretrained LMs via dual-encoder net- works (Step 1, §2.1), and then extracting the represen- tations from their (fine-tuned) layers (Step 2, §2.2). both static and contextualized WEs ultimately learn solely from the distributional word co-occurrence signal. This source of signal is known to lead to distortions in the induced representations by con- flating meaning based on topical relatedness rather than authentic semantic similarity (Hill et al., 2015; Schwartz et al., 2015; Vuli´c et al., 2017). This also creates a ripple effect on downstream applications, where model …

Content truncated to 500 words