How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speech

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 9370–9393 July 9-14, 2023 ©2023 Association for Computational Linguistics How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speech Aditya Yedetore∗1, Tal Linzen2, Robert Frank3, R. Thomas McCoy∗4 1Boston University, 2New York University, 3Yale University, 4Princeton University yedetore@bu.edu, linzen@nyu.edu, robert.frank@yale.edu, tom.mccoy@princeton.edu Abstract When acquiring syntax, children consistently choose hierarchical rules over competing non- hierarchical possibilities. Is this preference due to a learning bias for hierarchical struc- ture, or due to more general biases that interact with hierarchical cues in children’s linguistic input? We explore these possibilities by train- ing LSTMs and Transformers—two types of neural networks without a hierarchical bias— on data similar in quantity and content to chil- dren’s linguistic input: text from the CHILDES corpus. We then evaluate what these models have learned about English yes/no questions, a phenomenon for which hierarchical structure is crucial. We find that, though they perform well at capturing the surface statistics of child- directed speech (as measured by perplexity), both model types generalize in a way more con- sistent with an incorrect linear rule than the correct hierarchical rule. These results sug- gest that human-like generalization from text alone requires stronger biases than the general sequence-processing biases of standard neural network architectures. 1 Introduction Syntax is driven by hierarchical structure, yet we typically encounter sentences as linear sequences of words. How do children come to recognize the hierarchical nature of the languages they acquire? Some argue that humans must have a hierarchical inductive bias—an innate predisposition for hierar- chical structure (Chomsky, 1965, 1980). An alter- native view (e.g., Lewis and Elman, 2001) is that no such bias is necessary: there may be clear evi- dence for hierarchical structure in children’s input, so that children would choose hierarchical rules even without a hierarchical bias. ∗ Work done while at Johns Hopkins University. At first blush, recent work in natural language processing (NLP) may seem to indicate that no hier- archical bias is necessary. Neural networks trained on naturally-occurring text perform impressively on syntactic evaluations even though they have no explicit syntactic structure built into them (e.g., Gu- lordava et al., 2018; Wilcox et al., 2018; Warstadt et al., 2020a). However, these results do not pro- vide strong evidence about the learning biases re- quired to learn language from the data available to humans because these models receive very dif- ferent training data than humans do (Warstadt and Bowman, 2022). First, NLP models are typically trained on far more data than children receive, so models have more opportunities to encounter rare syntactic structures (Linzen, 2020). Second, most training sets in NLP are built from Internet text (e.g., Wikipedia), which differs qualitatively from the utterances that children typically hear; e.g., sen- tences in Wikipedia are on average 25 words long (Yasseri et al., 2012), compared to 5 words for sentences in the North American English subset of the CHILDES corpus of child-directed speech …

Content truncated to 500 words