Pretrained Language Model Embryology: The Birth of ALBERT

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6813–6828, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6813 Pretrained Language Model Embryology: The Birth of ALBERT Cheng-Han Chiang National Taiwan University, Taiwan dcml0714@gmail.com Sung-Feng Huang National Taiwan University, Taiwan f06942045@ntu.edu.tw Hung-yi Lee National Taiwan University, Taiwan hungyilee@ntu.edu.tw Abstract While behaviors of pretrained language mod- els (LMs) have been thoroughly examined, what happened during pretraining is rarely studied. We thus investigate the developmen- tal process from a set of randomly initialized parameters to a totipotent1 language model, which we refer to as the embryology of a pre- trained language model. Our results show that ALBERT learns to reconstruct and pre- dict tokens of different parts of speech (POS) in different learning speeds during pretrain- ing. We also find that linguistic knowledge and world knowledge do not generally improve as pretraining proceeds, nor do downstream tasks’ performance. These findings suggest that knowledge of a pretrained model varies during pretraining, and having more pretrain steps does not necessarily provide a model with more comprehensive knowledge. We pro- vide source codes and pretrained models to reproduce our results at https://github. com/d223302/albert-embryology. 1 Introduction The world of NLP has gone through some tremen- dous revolution since the proposal of contextual- ized word embeddings. Some big names are ELMo (Peters et al., 2018), GPT (Radford et al.), and BERT (Devlin et al., 2019), along with its vari- ants (Sanh et al., 2019; Liu et al., 2019b; Lan et al., 2019). Performance boosts on miscellaneous down- stream tasks have been reported by finetuning these totipotent pretrained language models. With a view to better grasping what has been learned by these contextualized word embedding models, probing is generally applied to the pretrained models and the 1According to Wikipedia, totipotency is the ability of a single cell to divide and produce all of the differentiated cells in an organism. We use its adjective form here to refer to the ability of a pretrained model which can be finetuned for a variety of downstream tasks. models finetuned from them. Probing targets can range from linguistic knowledge, including seman- tic roles and syntactic structures (Liu et al., 2019a; Tenney et al., 2019, 2018; Hewitt and Manning, 2019), to world knowledge (Petroni et al., 2019). While the previous work focuses on what knowledge has been learned after pretraining of transformer-based language models, few delve into their dynamics during pretraining. What happened during the training process of a deep neural net- work model has been widely studied, including Gur-Ari et al. (2018), Frankle et al. (2019), Raghu et al. (2017), Morcos et al. (2018). Some previ- ous works also study the dynamics of the training process of an LSTM language model (Saphra and Lopez, 2018, 2019), but the training dynamics of a large scale pretrained language models are not well-studied. In this work, we probe ALBERT (Lan et al., 2019) during its pretraining phase every N parameter update steps and study what it has learned and what it …

Content truncated to 500 words