Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1589 - 1598 July 10-15, 2022 ©2022 Association for Computational Linguistics Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models Karolina Sta´nczak1 Edoardo Ponti2 Lucas Torroba Hennigen3 Ryan Cotterell4 Isabelle Augenstein1 1University of Copenhagen 2Mila/McGill University 3Massachusetts Institute of Technology 4ETH Zürich ks@di.ku.dk augenstein@di.ku.dk edoardo-maria.ponti@mila.quebec lucastor@mit.edu rcotterell@inf.ethz.ch Abstract The success of multilingual pre-trained models is underpinned by their ability to learn representations shared by multiple languages even in absence of any explicit supervision. However, it remains unclear how these models learn to generalise across languages. In this work, we conjecture that multilingual pre- trained models can derive language-universal abstractions about grammar. In particular, we investigate whether morphosyntactic information is encoded in the same subset of neurons in different languages. We conduct the first large-scale empirical study over 43 languages and 14 morphosyntactic categories with a state-of-the-art neuron-level probe. Our findings show that the cross-lingual overlap between neurons is significant, but its extent may vary across categories and depends on language proximity and pre-training data size. https://github.com/copenlu/ multilingual-typology-probing 1 Introduction Massively multilingual pre-trained models (Devlin et al., 2019; Conneau et al., 2020; Liu et al., 2020; Xue et al., 2021, inter alia) display an impressive ability to transfer knowledge between languages as well as to perform zero-shot learning (Pires et al., 2019; Wu and Dredze, 2019; Nooralahzadeh et al., 2020; Hardalov et al., 2022, inter alia). Never- theless, it remains unclear how pre-trained models actually manage to learn multilingual representa- tions despite the lack of an explicit signal through parallel texts. Hitherto, many have speculated that the overlap of sub-words between cognates in re- lated languages plays a key role in the process of multilingual generalisation (Wu and Dredze, 2019; Cao et al., 2020; Pires et al., 2019; Abend et al., 2015; Vuli´c et al., 2020). Animacy Aspect Case Definiteness Gender Mood Number Person Tense 0 0.2 0.4 0.6 Morphosyntactic Categories Overlap Percentage Figure 1: Percentages of neurons most associated with a particular morphosyntactic category that overlap be- tween pairs of languages. Colours in the plot refer to 2 models: m-BERT (red) and XLM-R-base (blue). In this work, we offer a concurrent hypothesis to explain the multilingual abilities of various pre- trained models; namely, that they implicitly align morphosyntactic markers that fulfil a similar gram- matical function across languages, even in absence of any lexical overlap. More concretely, we conjec- ture that they employ the same subset of neurons to encode the same morphosyntactic information (such as gender for nouns and mood for verbs).1 To test the aforementioned hypothesis, we employ Sta´nczak et al.’s (2022) latent variable probe to identify the relevant subset of neurons in each lan- guage and then measure their cross-lingual overlap. We experiment with two multilingual pre-trained models, m-BERT (Devlin et al., 2019) and XLM- R (Conneau et al., 2020), probing them for mor- phosyntactic information in 43 languages from Uni- versal Dependencies (Nivre et al., …

Content truncated to 500 words