Probing Multilingual BERT for Genetic and Typological Signals

Proceedings of the 28th International Conference on Computational Linguistics, pages 1214–1228 Barcelona, Spain (Online), December 8-13, 2020 1214 Probing Multilingual BERT for Genetic and Typological Signals Taraka Rama University of North Texas taraka.kasi@gmail.com Lisa Beinborn VU Amsterdam l.m.beinborn@vu.nl Steffen Eger TU Darmstadt eger@aiphes.tu-darmstadt.de Abstract We probe the layers in multilingual BERT (mBERT) for phylogenetic and geographic language signals across 100 languages and compute language distances based on the mBERT representa- tions. We 1) employ the language distances to infer and evaluate language trees, finding that they are close to the reference family tree in terms of quartet tree distance, 2) perform distance matrix regression analysis, finding that the language distances can be best explained by phylogenetic and worst by structural factors and 3) present a novel measure for measuring diachronic meaning stability (based on cross-lingual representation variability) which correlates significantly with published ranked lists based on linguistic approaches. Our results contribute to the nascent field of typological interpretability of cross-lingual text representations. 1 Introduction Cross-lingual text representations have become extremely popular in NLP, since they promise universal text processing in multiple human languages with labeled training data only in a single one. They go back at least to the work of Klementiev et al. (2012), and have seen an exploding number of contributions in recent years. Recent cross-lingual models provide representations for about 100 languages and vary in their training objectives. In offline learning, cross-lingual representations are obtained by projecting independently trained monolingual representations into a shared representational space using bilingual lexical resources (Faruqui and Dyer, 2014; Artetxe et al., 2017). In joint learning (Wang et al., 2020), the cross-lingual representations are learned directly, for example as a byproduct of large-scale machine translation (Artetxe and Schwenk, 2018). As parallel data is scarce for less frequent language pairs, the multilingual BERT model (mBERT) simply trains the BERT architecture (Devlin et al., 2019) on multilingual input from Wikipedia. The cross-lingual signal is thus only learned implicitly because mBERT uses the same representational space independent of the input language. This naive approach yields surprisingly high scores for cross-lingual downstream tasks, but the transfer does not work equally well for all languages. Pires et al. (2019) show that the performance differences between languages are gradual and that the representational similarity between languages seem to correlate with typological features. These relationships between languages remain opaque in cross-lingual representations and pose a challenge for the evaluation of their adequacy. Evaluations in down-stream tasks are an unreliable approximation because they can often be solved with- out accounting for deep linguistic knowledge or for interdependencies between subgroups of languages (Liang et al., 2020). While more language-agnostic representations can be beneficial to improve the average performance in task-oriented settings and to smooth the performance differences between high- and low-resource languages (Libovick`y et al., 2019; Zhao et al., 2020a), linguists are more interested in the representational differences between languages. The field of computational historical linguistics, for example, examines subtle semantic and syntactic cues to infer phylogenetic relations between languages (Rama …

Content truncated to 500 words