Recurrent Neural Networks in Linguistic Theory: Revisiting Pinker and Prince (1988) and the Past Tense Debate

Recurrent Neural Networks in Linguistic Theory: Revisiting Pinker and Prince (1988) and the Past Tense Debate Christo Kirov Johns Hopkins University Baltimore, MD ckirov1@jhu.edu Ryan Cotterell Johns Hopkins University Baltimore, MD ryan.cotterell@jhu.edu Abstract Can advances in NLP help advance cog- nitive modeling? We examine the role of artificial neural networks, the current state of the art in many common NLP tasks, by returning to a classic case study. In 1986, Rumelhart and McClelland famously in- troduced a neural architecture that learned to transduce English verb stems to their past tense forms. Shortly thereafter, in 1988, Pinker and Prince presented a comprehen- sive rebuttal of many of Rumelhart and McClelland’s claims. Much of the force of their attack centered on the empirical in- adequacy of the Rumelhart and McClelland model. Today, however, that model is severely outmoded. We show that the Encoder-Decoder network architectures used in modern NLP systems obviate most of Pinker and Prince’s criticisms without requir- ing any simplification of the past tense map- ping problem. We suggest that the empirical performance of modern networks warrants a reexamination of their utility in linguistic and cognitive modeling. 1 Introduction In their famous 1986 opus, Rumelhart and McClelland (R&M) describe a neural network capa- ble of transducing English verb stems to their past tense. The strong cognitive claims in the article fomented a veritable brouhaha in the linguistics community and eventually led to the highly influ- ential rebuttal of Pinker and Prince (1988) (P&P). P&P highlighted the extremely poor empirical per- formance of the R&M model, and pointed out a number of theoretical issues with the model, which they suggested would apply to any neural network, contemporarily branded connectionist approaches. Their critique was so successful that many linguists and cognitive scientists to this day do not consider neural networks a viable approach to modeling lin- guistic data and human cognition. In the field of natural language processing (NLP), however, neural networks have experienced a renaissance. With novel architectures, large new data sets available for training, and access to exten- sive computational resources, neural networks now constitute the state of the art in many NLP tasks. However, NLP as a discipline has a distinct prac- tical bent and more often concerns itself with the large-scale engineering applications of language technologies. As such, the field’s findings are not always considered relevant to the scientific study of language (i.e., the field of linguistics). Recent work, however, has indicated that this perception is changing, with researchers, for example, prob- ing the ability of neural networks to learn syntactic dependencies like subject–verb agreement (Linzen et al., 2016). Moreover, in the domains of morphology and phonology, both NLP practitioners and linguists have considered virtually identical problems, seem- ingly unbeknownst to each other. For example, both computational and theoretical morphologists are concerned with how different inflected forms in the lexicon are related and how one can learn to generate such inflections from data. Indeed, the original R&M network focuses on such a gener- ation task, namely, generating English …

Content truncated to 500 words