Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2089–2105 May 2-6, 2023 ©2023 Association for Computational Linguistics Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders Ivan Vuli´c1 Goran Glavaš2 Fangyu Liu1 Nigel Collier1 Edoardo Maria Ponti3,1 Anna Korhonen1 1Language Technology Lab, TAL, University of Cambridge 2CAIDAS, University of Würzburg 3EdinburghNLP, University of Edinburgh {iv250, alk23}@cam.ac.uk Abstract Pretrained multilingual language models (LMs) can be successfully transformed into multilin- gual sentence encoders (SEs; e.g., LABSE, XMPNET) via additional fine-tuning or model distillation with parallel data. However, it re- mains unclear how to best leverage them to rep- resent sub-sentence lexical items (i.e., words and phrases) in cross-lingual lexical tasks. In this work, we probe SEs for the amount of cross-lingual lexical knowledge stored in their parameters, and compare them against the orig- inal multilingual LMs. We also devise a sim- ple yet efficient method for exposing the cross- lingual lexical knowledge by means of ad- ditional fine-tuning through inexpensive con- trastive learning that requires only a small amount of word translation pairs. Using bilin- gual lexical induction (BLI), cross-lingual lex- ical semantic similarity, and cross-lingual en- tity linking as lexical probing tasks, we report substantial gains on standard benchmarks (e.g., +10 Precision@1 points in BLI). The results indicate that the SEs such as LABSE can be ‘rewired’ into effective cross-lingual lexical en- coders via the contrastive learning procedure, and that it is possible to expose more cross- lingual lexical knowledge compared to using them as off-the-shelf SEs. This way, we also provide an effective tool for harnessing ‘covert’ multilingual lexical knowledge hidden in mul- tilingual sentence encoders. 1 Introduction Transfer learning with pretrained Language Mod- els (LMs) such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) offers unmatched per- formance in many NLP tasks (Wang et al., 2019; Raffel et al., 2020). However, despite the wealth of semantic knowledge stored in the pretrained LMs (Rogers et al., 2020; Vuli´c et al., 2020b), they do not produce coherent and effective sen- tence representations when used off-the-shelf (Liu et al., 2021c). To this effect, further specializa- tion for sentence-level semantics – not unlike the standard task fine-tuning – is needed (Reimers and Gurevych, 2019; Li et al., 2020; Yan et al., 2021, inter alia). LMs get transformed into sentence encoders (SEs) via dual-encoder frameworks that leverage contrastive learning objectives (van den Oord et al., 2018; Musgrave et al., 2020), in super- vised (i.e., leveraging labeled external data such as NLI or sentence similarity annotations) (Reimers and Gurevych, 2019; Vuli´c et al., 2021b; Liu et al., 2021a) or, more recently, fully unsupervised fine- tuning (Liu et al., 2021c; Gao et al., 2021) setups. Following the procedures from monolingual se- tups, another line of research has been transform- ing multilingual LMs into multilingual SEs (Feng et al., 2022; Reimers and Gurevych, 2020), which enable effective sentence matching and ranking in multiple languages as well as cross-lingually (Litschko et al., 2022). The transformation is typically …

Content truncated to 500 words