What do RNN Language Models Learn about Filler–Gap Dependencies?

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 211–221 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 211 What do RNN Language Models Learn about Filler–Gap Dependencies? Ethan Wilcox1, Roger Levy2, Takashi Morita3,4, and Richard Futrell5 1Department of Linguistics, Harvard University, wilcoxeg@g.harvard.edu 2Department of Brain and Cognitive Sciences, MIT, rplevy@mit.edu 3Primate Research Institute, Kyoto University, tmorita@alum.mit.edu 4Department of Linguistics and Philosophy, MIT 5Department of Language Science, UC Irvine, rfutrell@uci.edu Abstract RNN language models have achieved state- of-the-art perplexity results and have proven useful in a suite of NLP tasks, but it is as yet unclear what syntactic generalizations they learn. Here we investigate whether state-of- the-art RNN language models represent long- distance filler–gap dependencies and con- straints on them. Examining RNN behavior on experimentally controlled sentences de- signed to expose filler–gap dependencies, we show that RNNs can represent the relation- ship in multiple syntactic positions and over large spans of text. Furthermore, we show that RNNs learn a subset of the known restric- tions on filler–gap dependencies, known as is- land constraints: RNNs show evidence for wh-islands, adjunct islands, and complex NP islands. These studies demonstrates that state- of-the-art RNN models are able to learn and generalize about empty syntactic positions. 1 Introduction Many recent advancements in Natural Language Processing have come from the introduction of Recurrent Neural Networks (RNN) (Elman, 1990; Goldberg, 2017). One class of RNNs, the Long Short-Term Memory RNN (LSTM) (Hochreiter and Schmidhuber, 1997) has been able to achieve impressive results on a suite of NLP tasks, includ- ing machine translation, language modeling, and syntactic parsing (Sutskever et al., 2014; Vinyals et al., 2015; Jozefowicz et al., 2016). But the na- ture of the representations learned by these mod- els is not properly understood. As these models are being deployed with increasing frequency, this poses both engineering, accountability, and theo- retical problems. One promising line of research aims to crack open these ‘black boxes’ by investigating how LSTM language models perform on specially con- trolled sentences designed to draw out behavior that indicates representation of a syntactic depen- dency. Using this method, Linzen et al. (2016) and Gulordava et al. (2018) demonstrated that these models are able to successfully learn the number agreement dependency between a subject and its verb, even when there are intervening elements, and McCoy et al. (2018) found that RNNs learn the hierarchical rules of English auxiliary inver- sion. In this paper, we broaden and deepen this line of inquiry by examining what LSTMs learn about an unexplored syntactic relationship: the filler–gap dependency. The filler–gap dependency is novel, insofar as learning it requires the network to gen- eralize about the absence of material. For our purposes, filler–gap dependency refers to a relationship between a filler, which is a wh- complementizer such as ‘what’ or ‘who’, and a gap, which is an empty syntactic position licensed by the filler. In example (1a), the filler is ‘what’ and the gap appears after ‘devoured’, indicated with …

Content truncated to 500 words