On the Transformation of Latent Space in Fine-Tuned NLP Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1495 - 1516 December 7-11, 2022 ©2022 Association for Computational Linguistics On the Transformation of Latent Space in Fine-Tuned NLP Models WARNING: This paper contains model outputs which may be disturbing to the reader Nadir Durrani♢ Hassan Sajjad♣∗ Fahim Dalvi♢ Firoj Alam♢ ♢Qatar Computing Research Institute, Hamad Bin Khalifa University, Qatar ♣Faculty of Computer Science, Dalhousie University, Canada {ndurrani,faimaduddin, fialam}@hbku.edu.qa, hsajjad@dal.ca Abstract We study the evolution of latent space in fine- tuned NLP models. Different from the com- monly used probing-framework, we opt for an unsupervised method to analyze represen- tations. More specifically, we discover latent concepts in the representational space using hi- erarchical clustering. We then use an alignment function to gauge the similarity between the latent space of a pre-trained model and its fine- tuned version. We use traditional linguistic con- cepts to facilitate our understanding and also study how the model space transforms towards task-specific information. We perform a thor- ough analysis, comparing pre-trained and fine- tuned models across three models and three downstream tasks. The notable findings of our work are: i) the latent space of the higher lay- ers evolve towards task-specific concepts, ii) whereas the lower layers retain generic con- cepts acquired in the pre-trained model, iii) we discovered that some concepts in the higher layers acquire polarity towards the output class, and iv) that these concepts can be used for gen- erating adversarial triggers. 1 Introduction The revolution of deep learning models in NLP can be attributed to transfer learning from pre-trained language models. Contextualized representations learned within these models capture rich linguis- tic knowledge that can be leveraged towards novel tasks e.g. classification of COVID-19 tweets (Alam et al., 2021; Valdes et al., 2021), disease prediction (Rasmy et al., 2020) or natural language under- standing tasks such as SQUAD (Rajpurkar et al., 2016) and GLUE (Wang et al., 2018). Despite their success, the opaqueness of deep neural networks remain a cause of concern and has spurred a new area of research to analyze these models. A large body of work analyzed the knowl- edge learned within representations of pre-trained ∗ This work was carried out while the author was at QCRI. models (Belinkov et al., 2017; Conneau et al., 2018; Liu et al., 2019; Tenney et al., 2019; Durrani et al., 2019; Rogers et al., 2020) and showed the pres- ence of core-linguistic knowledge in various parts of the network. Although transfer learning using pre-trained models has become ubiquitous, very few papers (Merchant et al., 2020; Mosbach et al., 2020; Durrani et al., 2021) have analyzed the rep- resentations of the fine-tuned models. Given their massive usability, interpreting fine-tuned models and highlighting task-specific peculiarities is crit- ical for their deployment in real-word scenarios, where it is important to ensure fairness and trust when applying AI solutions. In this paper, we focus on analyzing fine-tuned models and investigate: how does the latent space evolve in a fine-tuned model? Different from …

Content truncated to 500 words