Recurrent babbling: evaluating the acquisition of grammar from limited input data

Proceedings of the 24th Conference on Computational Natural Language Learning, pages 165–176 Online, November 19-20, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 165 Recurrent babbling: evaluating the acquisition of grammar from limited input data Ludovica Pannitto CIMeC University of Trento ludovica.pannitto@unitn.it Aur´elie Herbelot CIMeC/DISI University of Trento aurelie.herbelot@unitn.it Abstract Recurrent Neural Networks (RNNs) have been shown to capture various aspects of syntax from raw linguistic input. In most previous ex- periments, however, learning happens over un- realistic corpora, which do not reflect the type and amount of data a child would be exposed to. This paper remedies this state of affairs by training a Long Short-Term Memory net- work (LSTM) over a realistically sized subset of child-directed input. The behaviour of the network is analysed over time using a novel methodology which consists in quantifying the level of grammatical abstraction in the model’s generated output (its ‘babbling’), compared to the language it has been exposed to. We show that the LSTM indeed abstracts new structures as learning proceeds. 1 Do RNNs learn grammar? Artificial Neural Networks, and Long Short-Term Memory Networks more specifically, have consis- tently demonstrated great capabilities in the area of language modeling. In addition to generating credible surface patterns, they show excellent per- formances when tested on very specific grammati- cal abilities (Gulordava et al., 2018; Lakretz et al., 2019), without requiring any prior bias towards the syntactic structure of natural languages. From a theoretical point of view, these results seem to contradict the well-known argument of the poverty of the stimulus (Chomksy, 1959; Chomsky, 1968) and raise questions about the continuity hy- pothesis in language acquisition (Lust, 1999; Crain and Pietroski, 2001). At the same time, a num- ber of results give a much more mitigated view of RNNs’ abstraction capabilities (Marvin and Linzen, 2018; Chowdhury and Zamparelli, 2018). It thus remains unclear how and to what extent grammati- cal abilities emerge in artificial language models, and how this knowledge is encoded in their repre- sentations – especially when considering notions such as productivity and compositionality (Baroni, 2020), which are recognised as defining traits of natural languages. This paper proposes that the evaluation of RNN grammars should be widened to include the effect of the type of input data fed to the network, as well as the theoretical paradigm used to analyse its output. We specifically remark that much of the dis- cussion concerning language modeling remains in- fluenced by the mainstream generativist approach, which posits a sharp distinction between syntax and the lexicon. Our own approach will be to de- part from this account by testing the grammatical abilities of an RNN in a usage-based perspective. Specifically, we ask what kind of structures are ab- stracted and used productively by the network, and how the abstraction process takes place over time. In contrast with previous models: (i) we train a vanilla char-LSTM on a more realistic variety and amount of data, focusing on a limited amount of child-directed language; (ii) we do not rely on …

Content truncated to 500 words