Training Trajectories of Language Models Across Scales

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13711–13738 July 9-14, 2023 ©2023 Association for Computational Linguistics Training Trajectories of Language Models Across Scales Mengzhou Xia1, Mikel Artetxe2, Chunting Zhou2, Xi Victoria Lin2, Ramakanth Pasunuru2, Danqi Chen1, Luke Zettlemoyer2, Ves Stoyanov2 1Princeton University 2Meta AI mengzhou@princeton.edu Abstract Scaling up language models has led to unprece- dented performance gains, but little is under- stood about how the training dynamics change as models get larger. How do language mod- els of different sizes learn during pre-training? Why do larger language models demonstrate more desirable behaviors? In this paper, we analyze the intermediate training checkpoints of differently sized OPT models (Zhang et al., 2022)—from 125M to 175B parameters—on next-token prediction, sequence-level genera- tion and downstream tasks. We find that 1) at a given perplexity and independent of model sizes, a similar subset of training tokens see the most significant reduction in loss, with the rest stagnating or showing double-descent be- havior (Nakkiran et al., 2020); 2) early in train- ing, all models learn to reduce the perplexity of grammatical sequences that contain halluci- nations, with small models halting at this sub- optimal distribution and larger ones eventually learning to assign these sequences lower prob- abilities; and 3) perplexity is a strong predic- tor of in-context learning performance on 74 multiple-choice tasks from BIG-Bench, and this holds independently of the model size. Together, these results show that perplexity is more predictive of model behaviors than model size or training computation.1 1 Introduction Scaling up language models has been shown to im- prove language modeling perplexity (Kaplan et al., 2020; Hernandez et al., 2022) as well as zero- or few-shot end task accuracies (Brown et al., 2020; Rae et al., 2021; Chowdhery et al., 2022; Zhang et al., 2022). However, relatively little is under- stood about why or how this happens. How do the training dynamics differ as models get larger? What do language models of different sizes learn 1Code is publicly available at https://github.com/ xiamengzhou/training_trajectory_analysis. during pre-training in terms of both generating texts and solving end tasks? We attempt to make progress to answer these questions by studying the training trajectories of differently-sized OPT models (Zhang et al., 2022) through analyzing their intermediate checkpoints. In contrast to prior work, which studies the trajecto- ries of small models with up to 300M parameters (Liu et al., 2021; Choshen et al., 2022; Blevins et al., 2022) or focuses on the language modeling objective alone (Kaplan et al., 2020; Hernandez et al., 2021, 2022), we are the first to comprehen- sively study the training trajectories of large-scale autoregressive language models with up to 175B parameters across a wide range of settings. Repeatedly across training and different model scales, we analyze three aspects of model perfor- mance: (i) next-token prediction on subsets of to- kens (ii) sequence-level generation and (iii) down- stream task performance. We use perplexity, which is closely tied to language model evaluation, as the major metric …

Content truncated to 500 words