Are we there yet? Encoder-decoder neural networks as cognitive models of English past tense inflection

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3868–3877 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3868 Are we there yet? Encoder-decoder neural networks as cognitive models of English past tense inflection Maria Corkery mcorkery@inf.ed.ac.uk Yevgen Matusevych yevgen.matusevych@ed.ac.uk Sharon Goldwater sgwater@inf.ed.ac.uk School of Informatics University of Edinburgh Abstract The cognitive mechanisms needed to account for the English past tense have long been a subject of debate in linguistics and cognitive science. Neural network models were pro- posed early on, but were shown to have clear flaws. Recently, however, Kirov and Cotterell (2018) showed that modern encoder-decoder (ED) models overcome many of these flaws. They also presented evidence that ED mod- els demonstrate humanlike performance in a nonce-word task. Here, we look more closely at the behaviour of their model in this task. We find that (1) the model exhibits instabil- ity across multiple simulations in terms of its correlation with human data, and (2) even when results are aggregated across simulations (treating each simulation as an individual hu- man participant), the fit to the human data is not strong—worse than an older rule-based model. These findings hold up through sev- eral alternative training regimes and evaluation measures. Although other neural architectures might do better, we conclude that there is still insufficient evidence to claim that neural nets are a good cognitive model for this task. 1 Introduction For over 30 years, the English past tense has served as both inspiration and testbed for models of lan- guage acquisition and processing (Rumelhart and McClelland, 1986; Pinker and Prince, 1988; Mar- cus, 1995; Plunkett and Juola, 1999; Pinker and Ullman, 2002; Albright and Hayes, 2003; Seiden- berg and Plaut, 2014; Kirov and Cotterell, 2018; Blything et al., 2018, etc.). One of the most well- known debates centres on whether the apparently rule-governed regular past tense is indeed repre- sented cognitively using explicit rules. Rumelhart and McClelland (1986) famously argued against this hypothesis, presenting a neural network model intended to capture both regular and irregular verbs with no explicit rules. However, Pinker and Prince (1988) presented a scathing rebuttal, pointing out both theoretical and empirical failures of the model. In their alternative (dual-route) view, the regular past tense is categorical and captured via explicit rules, while irregular past tenses are memorized and can (occasionally) generalize via gradient ana- logical processes (Pinker and Prince, 1988; Prasada and Pinker, 1993). Their arguments were so influ- ential that although neural networks gained consid- erable traction in cognitive science more generally (Bechtel and Abrahamsen, 1991; McCloskey, 1991; Elman et al., 1996), many linguists dismissed the whole approach.1 With the recent success of deep learning in NLP, however, there has been renewed interest in ex- ploring the extent to which neural networks cap- ture human behaviour in psycholinguistic tasks (e.g., Linzen and Leonard, 2018; Linzen, 2019). In particular, Kirov and Cotterell (2018; henceforth K&C) revisited the past tense debate and showed that modern sequence-based encoder-decoder (ED) models …

Content truncated to 500 words