Cross-Linguistic Syntactic Difference in Multilingual BERT: How Good is It and How Does It Affect Transfer?

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8073 - 8092 December 7-11, 2022 ©2022 Association for Computational Linguistics Cross-Linguistic Syntactic Difference in Multilingual BERT: How Good is It and How Does It Affect Transfer? Ningyu Xu1,2, Tao Gui2∗, Ruotian Ma1, Qi Zhang1, Jingting Ye3,4, Menghan Zhang2, Xuanjing Huang1,2 1School of Computer Science, Fudan University 2Institute of Modern Languages and Linguistics, Fudan University 3Department of Chinese Language and Literature, Fudan University 4Department of Linguistic and Cultural Evolution, Max Planck Institute for Evolutionary Anthropology nyxu22@m.fudan.edu.cn {tgui,rtma19,qz,yejingting,mhzhang,xjhuang}@fudan.edu.cn Abstract Multilingual BERT (mBERT) has demon- strated considerable cross-lingual syntactic ability, whereby it enables effective zero-shot cross-lingual transfer of syntactic knowledge. The transfer is more successful between some languages, but it is not well understood what leads to this variation and whether it fairly reflects difference between languages. In this work, we investigate the distributions of gram- matical relations induced from mBERT in the context of 24 typologically different languages. We demonstrate that the distance between the distributions of different languages is highly consistent with the syntactic difference in terms of linguistic formalisms. Such difference learnt via self-supervision plays a crucial role in the zero-shot transfer performance and can be predicted by variation in morphosyntactic prop- erties between languages. These results suggest that mBERT properly encodes languages in a way consistent with linguistic diversity and provide insights into the mechanism of cross- lingual transfer. 1 Introduction Cross-lingual transfer aims to address the huge linguistic disparity in NLP by transferring the knowledge acquired in high-resource languages to low-resource ones, where pretrained multilingual encoders, such as Multilingual BERT (mBERT) (Devlin et al., 2019), have proven a powerful facilitator. Compared to other approaches learning certain cross-lingual alignment in a supervised (Gouws et al., 2015; Mikolov et al., 2013; Faruqui and Dyer, 2014) or unsupervised (Artetxe et al., 2017; Zhang et al., 2017; Lample et al., 2018) manner, mBERT directly learns to encode different languages in a shared representation space through self-supervised joint training, dispensing with explicit alignment. It has exhibited notable cross- lingual ability and can perform effective zero- ∗Corresponding author. shot cross-lingual transfer across a variety of downstream tasks, albeit the performance varies (Wu and Dredze, 2019; Pires et al., 2019). The simplicity and efficacy of mBERT are crucial for cross-lingual transfer and have sparked interest in investigating the reason for its success. Previous work has looked into its representation space and found that mBERT automatically per- forms certain alignment across languages (Cao et al., 2019; Gonen et al., 2020; Conneau et al., 2020; Chi et al., 2020). The extent of alignment is shown correlated with the transfer performance (Muller et al., 2021). Despite these insights into the source of the transfer, it is also intriguing why different languages are aligned to varying degrees and what implication such variation bears. Another line of work has demonstrated that the zero-shot transfer performance is affected by certain linguistic features such as word order (Pires et al., 2019; Karthikeyan et al., 2019), whereas the underlying …

Content truncated to 500 words