Examining Cross-lingual Contextual Embeddings with Orthogonal Structural Probes

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4589–4598 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4589 Examining Cross-lingual Contextual Embeddings with Orthogonal Structural Probes Tomasz Limisiewicz and David Mareˇcek Institute of Formal and Applied Linguistics, Faculty of Mathematics and Physics Charles University, Prague, Czech Republic {limisiewicz, marecek}@ufal.mff.cuni.cz Abstract State-of-the-art contextual embeddings are ob- tained from large language models available only for a few languages. For others, we need to learn representations using a multi- lingual model. There is an ongoing debate on whether multilingual embeddings can be aligned in a space shared across many lan- guages. The novel Orthogonal Structural Probe (Limisiewicz and Mareˇcek, 2021) al- lows us to answer this question for specific linguistic features and learn a projection based only on mono-lingual annotated datasets. We evaluate syntactic (UD) and lexical (WordNet) structural information encoded in MBERT’s contextual representations for nine diverse lan- guages.1 We observe that for languages closely related to English, no transformation is needed. The evaluated information is en- coded in a shared cross-lingual embedding space. For other languages, it is beneficial to apply orthogonal transformation learned sep- arately for each language. We successfully apply our findings to zero-shot and few-shot cross-lingual parsing. 1 Introduction The representation learned by language models has been successfully applied in various NLP tasks. Multilingual pre-training allows utilizing the rep- resentation for various languages, including low- resource ones. There is an open discussion about to what extent contextual embeddings are similar across languages (Søgaard et al., 2018; Hartmann et al., 2019; Vuli´c et al., 2020). The motivation for our work is to answer: Q1 Is linguistic information uniformly encoded in the representations of various languages? And if this assumption does not hold: Q2 Is it possible to learn orthogonal transformation to align the embeddings? 1English, Spanish, Slovene, Indonesian, Chinese, Finnish, Arabic, French, and Basque We probe for the syntactic and lexical structures encoded in multilingual embeddings with the new Orthogonal Structural Probes (Limisiewicz and Mareˇcek, 2021). Previously, Chi et al. (2020) em- ployed structural probing (Hewitt and Manning, 2019) to evaluate cross-lingual syntactic informa- tion in MBERT and visualize how it is distributed across languages. Our approach’s advantage is learning an orthogonal transformation that maps the embeddings across languages based on mono- lingual linguistic information: dependency syntax and lexical hypernymy. This new capability allows us to test different probing scenarios. We mea- sure how adding assumptions of isomorphism and uniformity of the representations across languages affect probing results to answer our research ques- tions. 2 Related Work Probing It is a method of evaluating linguistic information encoded in pre-trained NLP models. Usually, a simple classifier for the probing task is trained on the frozen model’s representation (Linzen et al., 2016; Belinkov et al., 2017; Blevins et al., 2018). The work of Hewitt and Manning (2019) introduced structural probes that linearly transform contextual embeddings to approximate the topology of dependency trees. Limisiewicz and Mareˇcek (2021) proposed new structural tasks and introduced orthogonal constraint allowing to …

Content truncated to 500 words