Priorless Recurrent Networks Learn Curiously

Proceedings of the 28th International Conference on Computational Linguistics, pages 5147–5158 Barcelona, Spain (Online), December 8-13, 2020 5147 Priorless Recurrent Networks Learn Curiously Jeff Mitchell School of Psychological Science University of Bristol Bristol, UK jeff.mitchell@bristol.ac.uk Jeffrey S. Bowers School of Psychological Science University of Bristol Bristol, UK j.bowers@bristol.ac.uk Abstract Recently, domain-general recurrent neural networks, without explicit linguistic inductive biases, have been shown to successfully reproduce a range of human language behaviours, such as ac- curately predicting number agreement between nouns and verbs. We show that such networks will also learn number agreement within unnatural sentence structures, i.e. structures that are not found within any natural languages and which humans struggle to process. These results suggest that the models are learning from their input in a manner that is substantially different from human language acquisition, and we undertake an analysis of how the learned knowledge is stored in the weights of the network. We find that while the model has an effective under- standing of singular versus plural for individual sentences, there is a lack of a unified concept of number agreement connecting these processes across the full range of inputs. Moreover, the weights handling natural and unnatural structures overlap substantially, in a way that underlines the non-human-like nature of the knowledge learned by the network. 1 Introduction Grammarians have for a long time (e.g.: P¯an.ini, 6th Century BCE; Aristotle, 350 BCE) been interested in characterising what is and is not possible within a given specific language. Famously, Chomsky (1957) used the sentence colorless green ideas sleep furiously and its reversal furiously sleep ideas green colorless to illustrate the difference between grammatical and ungrammatical within English. In addition, modern linguistics has also been concerned more generally with what is and is not possible within the space of all human languages. In this article, we investigate the ability of a neural language model to learn structures on the impossi- ble side of this boundary. That is, we train and test the network on data which has been manipulated to contain unnatural structures. In contrast, the evaluation of such models has usually focused on the other side of the boundary, i.e. on data gathered from real human behaviour within specific natural languages. Here, we are interested in the more general question of whether these neural architectures are appro- priate models of human language capacities, in terms of what differentiates possible from impossible structures. From a theoretical perspective, a number of grammatical formalisms have been proposed as compu- tational models of the space of possible natural languages: from Context-Free Grammars (Chomsky, 1956) to Tree-Adjoining Grammars (Joshi et al., 1969) and Combinatory Categorial Grammars (Steed- man, 1987). One characteristic that all such grammars share is the organisation of linguistic expressions into hierarchical, as opposed to purely linear, structures. Empirical research has also uncovered further design patterns common across many superficially di- verse languages. Although the concept of language universals is not uncontroversial, most linguist would agree that, at the very least, natural languages display strong …

Content truncated to 500 words