On the Nature of BERT: Correlating Fine-Tuning and Linguistic Competence

Proceedings of the 29th International Conference on Computational Linguistics, pages 3109–3119 October 12–17, 2022. 3109 On the Nature of BERT: Correlating Fine-Tuning and Linguistic Competence Federica Merendi⋆, Felice Dell’Orletta⋄, Giulia Venturi⋄ ⋆Department of Computer Science, University of Pisa ⋄Istituto di Linguistica Computazionale “Antonio Zampolli”, Pisa ItaliaNLP Lab – www.italianlp.it f.merendi@studenti.unipi.it, {name.surname}@ilc.cnr.it Abstract Several studies in the literature on the inter- pretation of Neural Language Models (NLM) focus on the linguistic generalization abilities of pre-trained models. However, little attention is paid to how the linguistic knowledge of the models changes during the fine-tuning steps. In this paper, we contribute to this line of re- search by showing to what extent a wide range of linguistic phenomena are forgotten across 50 epochs of fine-tuning, and how the preserved linguistic knowledge is correlated with the res- olution of the fine-tuning task. To this end, we considered a quite understudied task where lin- guistic information plays the main role, i.e. the prediction of the evolution of written language competence of native language learners. In ad- dition, we investigate whether it is possible to predict the fine-tuned NLM accuracy across the 50 epochs solely relying on the assessed linguistic competence. Our results are encour- aging and show a high relationship between the model’s linguistic competence and its ability to solve a linguistically-based downstream task. 1 Introduction In the last few years, interest in assessing the lin- guistic generalization abilities of Neural Language Models (NLMs) has given rise to numerous studies aimed at investigating how the models are able to encode different types of linguistic phenomena. To this end, the most widespread methodology is the probing classification approach (Conneau et al., 2018; Warstadt et al., 2019; Hewitt and Liang, 2019), which showed that NLMs are able to en- code a variety of language properties (Rogers et al., 2020). Even if most of the work is focused on the linguistic competence of pre-trained models, a complementary line of research is devoted to understanding whether and to what extent the exist- ing competence is modified by fine-tuning the pre- trained model on a supervised downstream dataset (Merchant et al., 2020). In this paper, we continue this line of research and carry out a study aimed at adopting a prob- ing task approach to investigate how the linguistic competence of one of the most prominent NLM model, BERT (Devlin et al., 2019), is altered after a fine-tuning stage with the main focus on which specific phenomena are preserved or forgotten and which fine-tuning epochs are mainly involved. As a testbed, we chose a downstream task where linguis- tic information plays the main role. It consists in predicting the evolution of written language com- petence relying on the linguistic style of chrono- logically ordered productions written by language learners (Richter et al., 2015; Miaschi et al., 2021a). To the best of our knowledge, it has been never con- sidered in the NLM interpretability literature, and differently from previous studies, the task was car- ried out at sentence rather than at …

Content truncated to 500 words