Language Model Decomposition: Quantifying the Dependency and Correlation of Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2508 - 2517 December 7-11, 2022 ©2022 Association for Computational Linguistics Language Model Decomposition: Quantifying the Dependency and Correlation of Language Models Hao Zhang Supportiv Inc, Berkeley, CA haozhang@alumni.princeton.edu Abstract Pre-trained language models (LMs), such as BERT (Devlin et al., 2018) and its variants, have led to significant improvements on various NLP tasks in past years. However, a theoretical framework for studying their relationships is still missing. In this paper, we fill this gap by investigating the linear dependency between pre-trained LMs. The linear dependency of LMs is defined analogously to the linear de- pendency of vectors. We propose Language Model Decomposition (LMD) to represent a LM using a linear combination of other LMs as basis, and derive the closed-form solution. A goodness-of-fit metric for LMD similar to the coefficient of determination is defined and used to measure the linear dependency of a set of LMs. In experiments, we find that BERT and eleven (11) BERT-like LMs are 91% lin- early dependent. This observation suggests that current state-of-the-art (SOTA) LMs are highly “correlated”. To further advance SOTA we need more diverse and novel LMs that are less de- pendent on existing LMs. 1 Introduction Large-scale pre-trained language models (LMs) have pushed state-of-the-art (SOTA) of NLP re- cently (Han et al., 2021; Qiu et al., 2020). Fol- lowing BERT (Devlin et al., 2018), many variants and improvements have been proposed since then. They differ from BERT in various aspects, in terms of training data (Liu et al., 2019), multilingual support (Conneau et al., 2019), model size (Sanh et al., 2019; Lan et al., 2019), pre-training objec- tive (Yang et al., 2019), model architecture (Clark et al., 2020), attention structure (He et al., 2020), and sequence length (Beltagy et al., 2020). How- ever, to the best of our knowledge, their relation- ships have not been studied from a mathematical perspective at the time of writing this paper. In this work, we present a quantitative frame- work for studying LM dependency and correlation. Conceptually, we view LM (e.g., encoders like BERT (Devlin et al., 2018)) as a vector-valued func- tion (or random vector) u(x) : Ω → Rd, where x is text sequence from the sequence space Ω, and Rd is the sequence embedding space. Sequence em- bedding is defined as the mean pooling of the last layer’s token embeddings. We define linear com- bination and dependency of LMs analogously to vectors/functions, and based upon these definitions we propose the Language Model Decomposition (LMD) algorithm to analyze the linear dependency of LMs. A goodness-of-fit metric is defined for LMD to quantify the linear dependency and corre- lation of LMs. In experiments, we find BERT and its successors are highly linearly dependent. Theoretically, the linear dependency between LMs implies redundancy, since they can be represented by each other. Practically, it simply means building more redundant LMs do not bring in new knowl- edge. If LMs are more linearly independent, we can …

Content truncated to 500 words