The emergence of number and syntax units in LSTM language models

Proceedings of NAACL-HLT 2019, pages 11–20 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 11 The Emergence of Number and Syntax Units in LSTM Language Models Yair Lakretz Cognitive Neuroimaging Unit NeuroSpin center 91191, Gif-sur-Yvette, France yair.lakretz@gmail.com German Kruszewski Facebook AI Research Paris, France germank@gmail.com Theo Desbordes Facebook AI Research Paris, France tdesbordes@fb.com Dieuwke Hupkes ILLC, University of Amsterdam Amsterdam, Netherlands d.hupkes@uva.nl Stanislas Dehaene Cognitive Neuroimaging Unit NeuroSpin center 91191, Gif-sur-Yvette, France stanislas.dehaene@gmail.com Marco Baroni Facebook AI Research Paris, France mbaroni@fb.com Abstract Recent work has shown that LSTMs trained on a generic language modeling objective capture syntax-sensitive generalizations such as long- distance number agreement. We have however no mechanistic understanding of how they ac- complish this remarkable feat. Some have conjectured it depends on heuristics that do not truly take hierarchical structure into account. We present here a detailed study of the inner mechanics of number tracking in LSTMs at the single neuron level. We discover that long- distance number information is largely man- aged by two “number units”. Importantly, the behaviour of these units is partially controlled by other units independently shown to track syntactic structure. We conclude that LSTMs are, to some extent, implementing genuinely syntactic processing mechanisms, paving the way to a more general understanding of gram- matical encoding in LSTMs. 1 Introduction In the last years, recurrent neural networks (RNNs), and particularly long-short-term-memory (LSTM) architectures (Hochreiter and Schmidhu- ber, 1997), have been successfully applied to a variety of NLP tasks. This has spurred interest in whether these generic sequence-processing de- vices are discovering genuine structural properties of language in their training data, or whether their success can be explained by opportunistic surface- pattern-based heuristics. Until now, this debate has mostly relied on “behavioural” evidence: The LSTM had been treated as a black box, and its capacities had been indirectly inferred by its performance on linguistic tasks. In this study, we took a com- plementary approach inspired by neuroscience: We thoroughly investigated the inner dynamics of an LSTM language model performing a number agreement task, striving to achieve a mechanis- tic understanding of how it accomplishes it. We found that the LSTM had specialized two “grand- mother” cells (Bowers, 2009) to carry number fea- tures from the subject to the verb across the in- tervening material.1 Interestingly, the LSTM also 1In the neuroscientific literature, “grandmother” cells are (sets of) neurons coding for specific information, e.g., about your grandmother, in a non-distributed manner. 12 possesses a more distributed mechanism to predict number when subject and verb are close, with the grandmother number cells only playing a crucial role in more difficult long-distance cases. Cru- cially, we independently identified a set of cells tracking syntactic structure, and found that one of them encodes the presence of an embedded phrase separating the main subject-verb depen- dency, and has strong efferent connections to the long-distance number cells, suggesting that the network relies on genuine syntactic information to regulate agreement-feature percolation. Our analysis thus provides direct evidence …

Content truncated to 500 words