The Grammar-Learning Trajectories of Neural Language Models

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8281 - 8297 May 22-27, 2022 c⃝2022 Association for Computational Linguistics The Grammar-Learning Trajectories of Neural Language Models Leshem Choshen†, Guy Hacohen†‡, Daphna Weinshall†, Omri Abend† Department of Computer Science† Department of Brain Sciences‡ Hebrew University of Jerusalem {first.last}@mail.huji.ac.il Abstract The learning trajectories of linguistic phe- nomena in humans provide insight into lin- guistic representation, beyond what can be gleaned from inspecting the behavior of an adult speaker. To apply a similar approach to analyze neural language models (NLM), it is first necessary to establish that different mod- els are similar enough in the generalizations they make. In this paper, we show that NLMs with different initialization, architecture, and training data acquire linguistic phenomena in a similar order, despite their different end per- formance. These findings suggest that there is some mutual inductive bias that underlies these models’ learning of linguistic phenom- ena. Taking inspiration from psycholinguis- tics, we argue that studying this inductive bias is an opportunity to study the linguistic repre- sentation implicit in NLMs. Leveraging these findings, we compare the rel- ative performance on different phenomena at varying learning stages with simpler reference models. Results suggest that NLMs exhibit consistent “developmental” stages. Moreover, we find the learning trajectory to be approxi- mately one-dimensional: given an NLM with a certain overall performance, it is possible to predict what linguistic generalizations it has al- ready acquired. Initial analysis of these stages presents phenomena clusters (notably morpho- logical ones), whose performance progresses in unison, suggesting a potential link between the generalizations behind them. 1 Introduction Children present remarkable consistency in their patterns of language acquisition. They often ac- quire linguistic phenomena in a similar order (Kuhl et al., 1992; Ingram, 1989), and make similar gen- eralizations and over-generalizations (Kuczaj II, 1977; Pinker, 1995). This consistency provides an important starting point for linguistic study. For example, arguments in favor of single or dual sys- tem accounts of morphological representation are often backed by computational models of children learning trajectories (e.g., Rumelhart and McClel- land, 1986; Pinker and Prince, 1988; Kirov and Cotterell, 2018). In this paper, we embrace this program for the study of computational language models, investigating learning trajectories. 1 The representations that language models (LM) acquire have been studied extensively, including studying their learning dynamics to improve train- ing (see §6). However, very little work aimed at drawing connections between the training dynam- ics and the learned representations. In this work we adopt a behavioral approach, thus revealing that NLMs share learning trajectories and generalize in similar ways during training. This implies that studying trajectories of NLMs is worthwhile, in the sense that results on one architecture or size are expected to be reproducible by others. These findings call for a characterization of these trajectories, a new and promising territory for re- search. We take first steps to explore these direc- tions, emphasizing their potential benefit to a better future understanding of …

Content truncated to 500 words