Finding Universal Grammatical Relations in Multilingual BERT

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5564–5577 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5564 Finding Universal Grammatical Relations in Multilingual BERT Ethan A. Chi, John Hewitt, and Christopher D. Manning Department of Computer Science Stanford University {ethanchi,johnhew,manning}@cs.stanford.edu Abstract Recent work has found evidence that Multi- lingual BERT (mBERT), a transformer-based multilingual masked language model, is capa- ble of zero-shot cross-lingual transfer, suggest- ing that some aspects of its representations are shared cross-lingually. To better under- stand this overlap, we extend recent work on finding syntactic trees in neural networks’ in- ternal representations to the multilingual set- ting. We show that subspaces of mBERT rep- resentations recover syntactic tree distances in languages other than English, and that these subspaces are approximately shared across languages. Motivated by these results, we present an unsupervised analysis method that provides evidence mBERT learns representa- tions of syntactic dependency labels, in the form of clusters which largely agree with the Universal Dependencies taxonomy. This evi- dence suggests that even without explicit su- pervision, multilingual masked language mod- els learn certain linguistic universals. 1 Introduction Past work (Liu et al., 2019; Tenney et al., 2019a,b) has found that masked language models such as BERT (Devlin et al., 2019) learn a surprising amount of linguistic structure, despite a lack of direct linguistic supervision. Recently, large mul- tilingual masked language models such as Multi- lingual BERT (mBERT) and XLM (Conneau and Lample, 2019; Conneau et al., 2019) have shown strong cross-lingual performance on tasks like XNLI (Lample and Conneau, 2019; Williams et al., 2018) and dependency parsing (Wu and Dredze, 2019). Much previous analysis has been motivated by a desire to explain why BERT-like models per- form so well on downstream applications in the monolingual setting, which begs the question: what properties of these models make them so cross- lingually effective? Figure 1: t-SNE visualization of head-dependent de- pendency pairs belonging to selected dependencies in English and French, projected into a syntactic subspace of Multilingual BERT, as learned on English syntax trees. Colors correspond to gold UD dependency type labels. Although neither mBERT nor our probe was ever trained on UD dependency labels, English and French dependencies exhibit cross-lingual clustering that largely agrees with UD dependency labels. In this paper, we examine the extent to which Multilingual BERT learns a cross-lingual repre- sentation of syntactic structure. We extend prob- ing methodology, in which a simple supervised model is used to predict linguistic properties from a model’s representations. In a key departure from past work, we not only evaluate a probe’s perfor- mance (on recreating dependency tree structure), but also use the probe as a window into understand- ing aspects of the representation that the probe was not trained on (i.e. dependency labels; Fig- ure 1). In particular, we use the structural prob- ing method of Hewitt and Manning (2019), which probes for syntactic trees by finding a linear trans- formation under which two words’ distance in their dependency …

Content truncated to 500 words