What’s so special about BERT’s layers? A closer look at the NLP pipeline in monolingual and multilingual models

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4339–4350 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4339 What’s so special about BERT’s layers? A closer look at the NLP pipeline in monolingual and multilingual models Wietse de Vries, Andreas van Cranenburgh, and Malvina Nissim CLCG, University of Groningen, The Netherlands {wietse.de.vries,a.w.van.cranenburgh,m.nissim}@rug.nl Abstract Peeking into the inner workings of BERT has shown that its layers resemble the classical NLP pipeline, with progressively more com- plex tasks being concentrated in later layers. To investigate to what extent these results also hold for a language other than English, we probe a Dutch BERT-based model and the mul- tilingual BERT model for Dutch NLP tasks. In addition, through a deeper analysis of part- of-speech tagging, we show that also within a given task, information is spread over different parts of the network and the pipeline might not be as neat as it seems. Each layer has different specialisations, so that it may be more useful to combine information from different layers, instead of selecting a single one based on the best overall performance. 1 Introduction and Background Natural Language Processing is now dominated by transformer-based models (Vaswani et al., 2017), like BERT (Devlin et al., 2019), a model trained on predicting masked tokens and relations between sentences. BERT’s impact is so strong that we al- ready talk about ‘BERTology’ (Rogers et al., 2020). In addition to using BERT in NLP tasks and end applications, research has also been done on BERT, especially to reveal what linguistic information is available in different parts of the model. This is done, e.g., investigating what BERT’s attention heads might be attending to (Clark et al., 2019), or looking at its internal vector representations using so-called probing (or diagnostic) classifiers (Ten- ney et al., 2019a). It has been noted that BERT pro- gressively acquires linguistic information roughly in the same the order of the classic language pro- cessing pipeline (Tenney et al., 2019b,a): surface features are expressed in lower layers, syntactic features more in middle layers and semantic ones in higher layers (Jawahar et al., 2019). So, for ex- ample, information on part-of-speech appears to be acquired earlier than on coreference. Most work dedicated to understanding the inner workings of BERT has focused on English, though non-English BERT models do exist, in two forms. One is a multilingual model (Devlin et al., 2019, mBERT), which is trained on Wikipedia dumps of 104 different languages. The other one is a series of monolingual BERTs (Polignano et al., 2019; Le et al., 2019; Virtanen et al., 2019; Martin et al., 2019; de Vries et al., 2019, among others). As ex- pected, also the non-English monolingual BERT models achieve state-of-the-art results on a variety of NLP tasks, and mostly outperform the multilin- gual model on common NLP tasks (Nozza et al., 2020). Nevertheless, mBERT performs surpris- ingly well on zero-shot POS tagging and Named Entity Recognition (NER), as well as on cross- lingual model transfer (Pires et …

Content truncated to 500 words