Combining Static and Contextualised Multilingual Embeddings

Findings of the Association for Computational Linguistics: ACL 2022, pages 2316 - 2329 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Combining Static and Contextualised Multilingual Embeddings Katharina Hämmerl1 and Jindˇrich Libovický1,2 and Alexander Fraser1 1Center for Information and Language Processing, LMU Munich, Germany 2Faculty of Mathematics and Physics, Charles University, Czech Republic {haemmerl,fraser}@cis.lmu.de libovicky@ufal.mff.cuni.cz Abstract Static and contextual multilingual embeddings have complementary strengths. Static em- beddings, while less expressive than contex- tual language models, can be more straight- forwardly aligned across multiple languages. We combine the strengths of static and con- textual models to improve multilingual rep- resentations. We extract static embeddings for 40 languages from XLM-R, validate those embeddings with cross-lingual word retrieval, and then align them using VecMap. This re- sults in high-quality, highly multilingual static embeddings. Then we apply a novel contin- ued pre-training approach to XLM-R, leverag- ing the high quality alignment of our static embeddings to better align the representation space of XLM-R. We show positive results for multiple complex semantic tasks. We release the static embeddings and the continued pre- training code.1 Unlike most previous work, our continued pre-training approach does not require parallel text. 1 Introduction Multilingual contextual encoders like XLM-R (Conneau et al., 2020a) and mBERT (Devlin et al., 2019), despite being trained without paral- lel data, exhibit “surprising” cross-linguality (Wu and Dredze, 2019; Conneau et al., 2020b) and have demonstrated strong performance on mul- tilingual and cross-lingual tasks (e.g., Hu et al., 2020; Lauscher et al., 2020; Kurfalı and Östling, 2021; Turc et al., 2021). However, their language- neutrality, meaning how well languages are aligned with each other, has clear limits (Libovický et al., 2020; Cao et al., 2020, inter alia). In particular, more typologically distant language pairs tend to be less well-aligned than more similar ones, affect- ing transfer performance. 1github.com/KathyHaem/combining-static- contextual X2S-MA VecMap X2S-M XLM-R <s> Die Katze ist gestreift. </s> a) chat X2S-MA XLM-R <s> chat </s> CCA/MSE b) Figure 1: a) We feed sentences from 40 monolingual corpora to XLM-R, extracting partially aligned mul- tilingual static embeddings (X2S-M). Then, we use VecMap to align the embeddings further, giving us X2S-MA. The German example sentence reads ‘The cat is striped’. b) Taking the representations of words from X2S-MA and XLM-R, we train the contextual model representations to be more similar to the well- aligned static embeddings via an alignment loss (CCA or MSE). The French example ‘chat’ means ‘cat’. By contrast, cross-lingual alignment is well- studied for static embeddings (e.g., Mikolov et al., 2013; Artetxe et al., 2018a; Vuli´c et al., 2020). They can be aligned using simple transformations, resulting in high quality multilingual embeddings. However, static embeddings are considerably less expressive than contextual models and have in many applications been superseded by them. This paper aims to combine the strengths of static and contextual models, and explore how they may benefit from each other. Our methods require no parallel corpus. Figure 1 shows a schematic of our two-part approach with an example: The words 2316 ‘Katze’ …

Content truncated to 500 words