Probing Pretrained Language Models for Lexical Semantics

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7222–7240, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7222 Probing Pretrained Language Models for Lexical Semantics Ivan Vuli´c♠ Edoardo M. Ponti♠ Robert Litschko♦ Goran Glavaˇs♦ Anna Korhonen♠ ♠Language Technology Lab, University of Cambridge, UK ♦Data and Web Science Group, University of Mannheim, Germany {iv250,ep490,alk23}@cam.ac.uk {goran,litschko}@informatik.uni-mannheim.de Abstract The success of large pretrained language mod- els (LMs) such as BERT and RoBERTa has sparked interest in probing their representa- tions, in order to unveil what types of knowl- edge they implicitly capture. While prior re- search focused on morphosyntactic, semantic, and world knowledge, it remains unclear to which extent LMs also derive lexical type-level knowledge from words in context. In this work, we present a systematic empirical anal- ysis across six typologically diverse languages and five different lexical tasks, addressing the following questions: 1) How do different lexi- cal knowledge extraction strategies (monolin- gual versus multilingual source LM, out-of- context versus in-context encoding, inclusion of special tokens, and layer-wise averaging) impact performance? How consistent are the observed effects across tasks and languages? 2) Is lexical knowledge stored in few parame- ters, or is it scattered throughout the network? 3) How do these representations fare against traditional static word vectors in lexical tasks? 4) Does the lexical information emerging from independently trained monolingual LMs dis- play latent similarities? Our main results in- dicate patterns and best practices that hold uni- versally, but also point to prominent variations across languages and tasks. Moreover, we val- idate the claim that lower Transformer layers carry more type-level lexical knowledge, but also show that this knowledge is distributed across multiple layers. 1 Introduction and Motivation Language models (LMs) based on deep Trans- former networks (Vaswani et al., 2017), pretrained on unprecedentedly large amounts of text, offer un- matched performance in virtually every NLP task (Qiu et al., 2020). Models such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019c), and T5 (Raffel et al., 2019) replaced task-specific neural architectures that relied on static word embeddings (WEs; Mikolov et al., 2013b; Pennington et al., 2014; Bojanowski et al., 2017), where each word is assigned a single (type-level) vector. While there is a clear consensus on the effec- tiveness of pretrained LMs, a body of recent re- search has aspired to understand why they work (Rogers et al., 2020). State-of-the-art models are “probed” to shed light on whether they capture task-agnostic linguistic knowledge and structures (Liu et al., 2019a; Belinkov and Glass, 2019; Ten- ney et al., 2019); e.g., they have been extensively probed for syntactic knowledge (Hewitt and Man- ning, 2019; Jawahar et al., 2019; Kulmizev et al., 2020; Chi et al., 2020, inter alia) and morphology (Edmiston, 2020; Hofmann et al., 2020). In this work, we put focus on uncovering and un- derstanding how and where lexical semantic knowl- edge is coded in state-of-the-art LMs. While pre- liminary findings from Ethayarajh (2019) and Vuli´c et al. (2020) suggest that there is …

Content truncated to 500 words