Towards General Natural Language Understanding with Probabilistic Worldbuilding

Word Acquisition in Neural Language Models Tyler A. Chang1,2, Benjamin K. Bergen1 1Department of Cognitive Science 2Halıcıo˘glu Data Science Institute University of California, San Diego, USA {tachang, bkbergen}@ucsd.edu Abstract We investigate how neural language mod- els acquire individual words during training, extracting learning curves and ages of acqui- sition for over 600 words on the MacArthur- Bates Communicative Development Inventory (Fenson et al., 2007). Drawing on studies of word acquisition in children, we evaluate mul- tiple predictors for words’ ages of acquisition in LSTMs, BERT, and GPT-2. We find that the effects of concreteness, word length, and lexical class are pointedly different in children and language models, reinforcing the impor- tance of interaction and sensorimotor experi- ence in child language acquisition. Language models rely far more on word frequency than children, but, like children, they exhibit slower learning of words in longer utterances. Inter- estingly, models follow consistent patterns during training for both unidirectional and bidi- rectional models, and for both LSTM and Transformer architectures. Models predict based on unigram token frequencies early in training, before transitioning loosely to bigram probabilities, eventually converging on more nuanced predictions. These results shed light on the role of distributional learning mecha- nisms in children, while also providing insights for more human-like language acquisition in language models. 1 Introduction Language modeling, predicting words from context, has grown increasingly popular as a pre- training task in NLP in recent years; neural lan- guage models such as BERT (Devlin et al., 2019), ELMo (Peters et al., 2018), and GPT (Brown et al., 2020) have produced state-of-the-art performance on a wide range of NLP tasks. There is now a sub- stantial amount of work assessing the linguistic information encoded by language models (Rogers et al., 2020); in particular, behavioral approaches from psycholinguistics and cognitive science have been applied to study language model predictions (Futrell et al., 2019; Ettinger, 2020). From a cognitive perspective, language models are of theoretical interest as distributional models of lan- guage, agents that learn exclusively from statistics over language (Boleda, 2020; Lenci, 2018). However, previous psycholinguistic studies of language models have nearly always focused on fully-trained models, precluding comparisons to the wealth of literature on human language acqui- sition. There are limited exceptions. Rumelhart and McClelland (1986) famously studied past tense verb form learning in phoneme-level neural networks during training, a study which was repli- cated in more modern character-level recurrent neural networks (Kirov and Cotterell, 2018). How- ever, these studies focused only on sub-word fea- tures. There remains a lack of research on language acquisition in contemporary language models, which encode higher level features such as syntax and semantics. As an initial step towards bridging the gap between language acquisition and language mod- eling, we present an empirical study of word acquisition during training in contemporary lan- guage models, including LSTMs, GPT-2, and BERT. We consider how variables such as word frequency, concreteness, and lexical class con- tribute to words’ ages of acquisition in language models. Each of our selected variables has …

Content truncated to 500 words