Colorless Green Recurrent Networks Dream Hierarchically

Proceedings of NAACL-HLT 2018, pages 1195–1205 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Colorless green recurrent networks dream hierarchically Kristina Gulordava∗ Department of Linguistics University of Geneva kristina.gulordava@unige.ch Piotr Bojanowski Facebook AI Research Paris bojanowski@fb.com Edouard Grave Facebook AI Research New York egrave@fb.com Tal Linzen Department of Cognitive Science Johns Hopkins University tal.linzen@jhu.edu Marco Baroni Facebook AI Research Paris mbaroni@fb.com Abstract Recurrent neural networks (RNNs) have achieved impressive results in a variety of lin- guistic processing tasks, suggesting that they can induce non-trivial properties of language. We investigate here to what extent RNNs learn to track abstract hierarchical syntactic struc- ture. We test whether RNNs trained with a generic language modeling objective in four languages (Italian, English, Hebrew, Russian) can predict long-distance number agreement in various constructions. We include in our evaluation nonsensical sentences where RNNs cannot rely on semantic or lexical cues (“The colorless green ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas I ate with the chair sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep furiously”), and, for Italian, we com- pare model performance to human intuitions. Our language-model-trained RNNs make re- liable predictions about long-distance agree- ment, and do not lag much behind human performance. We thus bring support to the hypothesis that RNNs are not just shallow- pattern extractors, but they also acquire deeper grammatical competence. 1 Introduction Recurrent neural networks (RNNs; Elman, 1990) are general sequence processing devices that do not explicitly encode the hierarchical structure that is thought to be essential to natural language (Everaert et al., 2015). Early work using ar- tificial languages showed that they may never- theless be able to approximate context-free lan- guages (Elman, 1991). More recently, RNNs have ∗The work was conducted during the internship at Face- book AI Research, Paris. achieved impressive results in large-scale tasks such as language modeling for speech recognition and machine translation, and are by now standard tools for sequential natural language tasks (e.g., Mikolov et al., 2010; Graves, 2012; Wu et al., 2016). This suggests that RNNs may learn to track grammatical structure even when trained on nois- ier natural data. The conjecture is supported by the success of RNNs as feature extractors for syntac- tic parsing (e.g., Cross and Huang, 2016; Kiper- wasser and Goldberg, 2016; Zhang et al., 2017). Linzen et al. (2016) directly evaluated the ex- tent to which RNNs can approximate hierarchi- cal structure in corpus-extracted natural language data. They tested whether RNNs can learn to predict English subject-verb agreement, a task thought to require hierarchical structure in the gen- eral case (“the girl girl girl girl girl girl girl girl girl girl girl girl girl girl girl girl girl the boys like... isisisisisisisisisisisisisisisisis or are are are are are are are are are are are are are are are are are?”). Their experiments confirmed that RNNs can, in principle, handle such constructions. However, in …

Content truncated to 500 words