A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5833–5847 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5833 A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space Alex Jones Dartmouth College alexander.g.jones.23@dartmouth.edu William Yang Wang University of California, Santa Barbara william@cs.ucsb.edu Kyle Mahowald University of Texas at Austin mahowald@utexas.edu Abstract In cross-lingual language models, representa- tions for many different languages live in the same space. Here, we investigate the linguistic and non-linguistic factors affecting sentence- level alignment in cross-lingual pretrained lan- guage models for 101 languages and 5,050 lan- guage pairs. Using BERT-based LaBSE and BiLSTM-based LASER as our models, and the Bible as our corpus, we compute a task- based measure of cross-lingual alignment in the form of bitext retrieval performance, as well as four intrinsic measures of vector space alignment and isomorphism. We then exam- ine a range of linguistic, quasi-linguistic, and training-related features as potential predictors of these alignment metrics. The results of our analyses show that word order agreement and agreement in morphological complexity are two of the strongest linguistic predictors of cross-linguality. We also note in-family train- ing data as a stronger predictor than language- specific training data across the board. We ver- ify some of our linguistic findings by looking at the effect of morphological segmentation on English-Inuktitut alignment, in addition to ex- amining the effect of word order agreement on isomorphism for 66 zero-shot language pairs from a different corpus. We make the data and code for our experiments publicly available.1 1 Introduction Cross-lingual language models are polyglots inso- far as they house representations for many different languages in the same space. But to what extent are they good polyglots? The answer depends, in part, on how well-aligned and isomorphic the representations are, and not all language pairs are equally well-aligned. What determines the qual- ity of the alignment? Are language pairs from the same family (e.g., Spanish and French) better 1https://github.com/AlexJonesNLP/ XLAnalysis5K Figure 1: A look at some of the strongest features for predicting cross-linguality, according to their num- ber of occurrences in best-feature regression searches across all our dependent variables (see Section 6.2). aligned than languages from two unrelated fami- lies (e.g., Japanese and Swahili)? Are languages which are geographically closer or share an alpha- bet better aligned? How do factors from linguistic typology (like word order and morphological mark- ing) affect alignment? Recent work has looked at the typological and training-related factors affecting cross-lingual alignment in monolingual embedding space (Vuli´c et al., 2020; Dubossarsky et al., 2020), assessed the cross-linguality of pretrained language models using probing tasks and downstream performance measures (Conneau et al., 2020; Wu and Dredze, 2019, 2020; Pires et al., 2019; Groenwold et al., 2020), and probed Transformer models (Wolf et al., 2020) for linguistic structure (see Rogers et al. 2020 for an overview of over 150 studies). However, a gap in the research exists regarding the following question: What are the linguistic, quasi-linguistic, and training-related factors determining the cross- linguality …

Content truncated to 500 words