Massively Multilingual Lexical Specialization of Multilingual Transformers

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7700–7715 July 9-14, 2023 ©2023 Association for Computational Linguistics Massively Multilingual Lexical Specialization of Multilingual Transformers Tommaso Green1, Simone Paolo Ponzetto1, Goran Glavaš2 1 Data and Web Science Group, University of Mannheim, Germany 2 CAIDAS, University of Würzburg, Germany {tommaso.green, ponzetto}@uni-mannheim.de goran.glavas@uni-wuerzburg.de Abstract While pretrained language models (PLMs) pri- marily serve as general-purpose text encoders that can be fine-tuned for a wide variety of downstream tasks, recent work has shown that they can also be rewired to produce high- quality word representations (i.e., static word embeddings) and yield good performance in type-level lexical tasks. While existing work primarily focused on the lexical specializa- tion of single monolingual PLMs, in this work we expose massively multilingual transformers (MMTs, e.g., mBERT or XLM-R) to multi- lingual lexical knowledge at scale, leveraging BabelNet as the readily available rich source of multilingual and cross-lingual type-level lex- ical knowledge. Concretely, we use Babel- Net’s multilingual synsets to create synonym pairs (or synonym-gloss pairs) across 50 lan- guages and then subject the MMTs (mBERT and XLM-R) to a lexical specialization pro- cedure guided by a contrastive objective. We show that such multilingual lexical specializa- tion brings substantial gains in two standard cross-lingual lexical tasks, bilingual lexicon in- duction and cross-lingual word similarity, as well as in cross-lingual sentence retrieval. Cru- cially, we observe gains for languages unseen in specialization, indicating that multilingual lexical specialization enables generalization to languages with no lexical constraints. In a se- ries of controlled experiments, we show that the number of specialization constraints plays a much greater role than the set of languages from which they originate. 1 Introduction Massively multilingual transfomers (MMTs) such as mBERT (Devlin et al., 2019) and XLM-R (Con- neau et al., 2020), among others, have been the primary vehicle of cross-lingual NLP transfer, of- fering state-of-the-art performance for many tasks and target languages in various zero-shot and few- shot transfer scenarios (Pires et al., 2019; Wu and Dredze, 2019; Cao et al., 2020; Artetxe et al., 2020; Lauscher et al., 2020a; Zhao et al., 2021; Ruder et al., 2021, inter alia). Much less work, however, investigated their capabilities as multilingual type- level word encoders (Vuli´c et al., 2020b). Recent work, focusing primarily on monolingual PLMs, demonstrated that they can be turned into effective type-level lexical encoders using lexical constraints (Vuli´c et al., 2021; Liu et al., 2021), i.e., a process commonly referred to as lexical specialization. Ex- isting work, however, investigated lexical special- ization in monolingual or bilingual settings only, namely specializing PLMs for a single language or a pair of languages using either English monolin- gual or noisy translated bilingual lexical constraints (Vuli´c et al., 2021). This is not only computation- ally inefficient, since one specialization needs to be executed for each language or language pair, but also does not tap into the wealth of multilingual knowledge of MMT’s simultaneous pretraining on many (100+) languages, as well as large …

Content truncated to 500 words