Analyzing Encoded Concepts in Transformer Language Models

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3082 - 3101 July 10-15, 2022 ©2022 Association for Computational Linguistics Analyzing Encoded Concepts in Transformer Language Models Hassan Sajjad⋄ Nadir Durrani⋄ Fahim Dalvi⋄ Firoj Alam⋄ Abdul Rafae Khan† Jia Xu† {hsajjad,ndurrani,faimaduddin,fialam}@hbku.edu.qa ⋄Qatar Computing Research Institute, HBKU Research Complex, Qatar {akhan4,jxu70}@stevens.edu †School of Engineering and Science, Steven Institute of Technology, USA Abstract We propose a novel framework ConceptX, to analyze how latent concepts are encoded in representations learned within pre-trained lan- guage models. It uses clustering to discover the encoded concepts and explains them by align- ing with a large set of human-defined concepts. Our analysis on seven transformer language models reveal interesting insights: i) the la- tent space within the learned representations overlap with different linguistic concepts to a varying degree, ii) the lower layers in the model are dominated by lexical concepts (e.g., affixa- tion), whereas the core-linguistic concepts (e.g., morphological or syntactic relations) are bet- ter represented in the middle and higher layers, iii) some encoded concepts are multi-faceted and cannot be adequately explained using the existing human-defined concepts.1 1 Introduction Contextualized word representations learned in deep neural network models (DDNs) capture rich concepts making them ubiquitous for transfer learn- ing towards downstream NLP. Despite their revolu- tion, the blackbox nature of the deep NLP models is a major bottle-neck for their large scale adapt- ability. Understanding the inner dynamics of these models is important to ensure fairness, robustness, reliability and control. A plethora of research has been carried out to probe DNNs for the linguistic knowledge (e.g. mor- phology, syntactic and semantic roles) captured within the learned representations. A commonly used framework to gauge how well linguistic infor- mation can be extracted from these models is the Probing Framework (Hupkes et al., 2018), where they train an auxiliary classifier using representa- tions as features to predict the property of inter- est. The performance of the classifier reflects the 1The code is available at https://github.com/ hsajjad/ConceptX. amount of knowledge learned within representa- tions. To this end, the researchers have analyzed what knowledge is learned within the representa- tions through relevant extrinsic phenomenon vary- ing from word morphology (Vylomova et al., 2016; Belinkov et al., 2017a) to high level concepts such as syntactic structure (Blevins et al., 2018; Marvin and Linzen, 2018) and semantics (Qian et al., 2016; Reif et al., 2019; Belinkov et al., 2017b) or more generic properties (Adi et al., 2016; Rogers et al., 2020). In this work, we approach the representation analysis from a different angle and present a novel framework ConceptX. In contrast to relying on the prediction capacity of the representations, we analyze the latent concepts learned within these rep- resentations and how knowledge is structured, us- ing an unsupervised method. More specifically, we question: i) do the representations encode knowl- edge inline with linguistic properties such as word morphology and semantics? ii) which properties dominate the overall structure in these representa- …

Content truncated to 500 words