An Isotropy Analysis in the Multilingual BERT Embedding Space

Findings of the Association for Computational Linguistics: ACL 2022, pages 1309 - 1316 May 22-27, 2022 c⃝2022 Association for Computational Linguistics An Isotropy Analysis in the Multilingual BERT Embedding Space Sara Rajaee⋆ ⋆ and Mohammad Taher Pilehvar⋆ ⋆ ⋆ ⋆ Iran University of Science and Technology, Tehran, Iran ⋆ ⋆ Tehran Institute for Advanced Studies, Khatam University, Iran sara_rajaee@comp.iust.ac.ir mp792@cam.ac.uk Abstract Several studies have explored various advan- tages of multilingual pre-trained models (such as multilingual BERT) in capturing shared lin- guistic knowledge. However, less attention has been paid to their limitations. In this pa- per, we investigate the multilingual BERT for two known issues of the monolingual mod- els: anisotropic embedding space and outlier dimensions. We show that, unlike its mono- lingual counterpart, the multilingual BERT model exhibits no outlier dimension in its rep- resentations while it has a highly anisotropic space. There are a few dimensions in the mono- lingual BERT with high contributions to the anisotropic distribution. However, we observe no such dimensions in the multilingual BERT. Furthermore, our experimental results demon- strate that increasing the isotropy of multilin- gual space can significantly improve its rep- resentation power and performance, similarly to what had been observed for monolingual CWRs on semantic similarity tasks. Our anal- ysis indicates that, despite having different de- generated directions, the embedding spaces in various languages tend to be partially similar with respect to their structures.1. 1 Introduction The multilingual BERT model (Devlin et al., 2019, mBERT), pre-trained on 104 languages with no supervision, has shown impressive ability in captur- ing linguistic knowledge across different languages (Pires et al., 2019). Many studies have explored the encoded knowledge in multilingual CWRs using probing tasks and under zero-shot setting (Wu and Dredze, 2019; K et al., 2020; Chi et al., 2020). Fol- lowing the probing studies, in this paper, we inves- tigate the multilingual embedding space of BERT, focusing on its geometry in terms of isotropy. Pre- vious research has shown that many pre-trained 1Our code and datasets are publicly available at: https://github.com/Sara-Rajaee/ Multilingual-Isotropy Figure 1: Degenerated (left) and isotropic (right) embed- ding spaces for Spanish plotted using PCA. Frequency- based distribution can be easily detected in the space (lighter colors indicate higher frequency). See Appendix C for more languages. models, such as GPT-2 (Radford et al., 2019), BERT, and RoBERTa (Liu et al., 2019), have de- generated embedding spaces that downgrade their semantic expressiveness (Ethayarajh, 2019; Cai et al., 2021; Rajaee and Pilehvar, 2021). Several proposals have been put forward to overcome this challenge (Gao et al., 2019; Zhang et al., 2020). However, to our knowledge, no study has so far been conducted on the degeneration problem in multilingual embedding spaces. Using two well-known metrics, we evaluate isotropy in the mBERT embedding space for six different languages (including two low resources): English, Spanish, Arabic, Turkish, Sundanese, and Swahili. We find that the representation spaces are massively anisotropic in all these languages. How- ever, unlike monolingual CWRs, where a few di- mensions dominate the cosine similarity …

Content truncated to 500 words