Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in Number and Gender Assignment

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 1–11 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 1 Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in Number and Gender Assignment Jaap Jumelet jumeletjaap@gmail.com University of Amsterdam Willem Zuidema w.h.zuidema@uva.nl ILLC, University of Amsterdam Dieuwke Hupkes d.hupkes@uva.nl ILLC, University of Amsterdam Abstract Extensive research has recently shown that re- current neural language models are able to pro- cess a wide range of grammatical phenomena. How these models are able to perform these remarkable feats so well, however, is still an open question. To gain more insight into what information LSTMs base their decisions on, we propose a generalisation of Contextual De- composition (GCD). In particular, this setup enables us to accurately distil which part of a prediction stems from semantic heuristics, which part truly emanates from syntactic cues and which part arise from the model biases themselves instead. We investigate this tech- nique on tasks pertaining to syntactic agree- ment and co-reference resolution and discover that the model strongly relies on a default rea- soning effect to perform these tasks. 1 Introduction Modern language models that use deep learn- ing architectures such as LSTMs, bi-LSTMs and Transformers, have shown enormous gains in per- formance in the last few years and are finding ap- plications in novel domains, ranging from speech recognition and writing assistance to autonomous generation of fake news. Understanding how they reach their predictions has become a key question for NLP, not only for purely scientific, but also for practical and ethical reasons. From a linguistic perspective, a natural ap- proach is to test the extent to which these models have learned classical linguistic constructs, such as inflectional morphology, constituency struc- ture, agreement between verb and subject, filler- gap dependencies, negative polarity or reflexive anaphora. An influential paper using this approach was presented by Linzen et al. (2016), who inves- tigated the performance of an LSTM-based lan- guage model on number agreement. In many later papers (e.g. Gulordava et al., 2018; Wilcox et al., 2018; Jumelet and Hupkes, 2018; Marvin and Linzen, 2018; Giulianelli et al., 2018) a wide spectrum of grammatical phenomena has been in- vestigated, assessing these grammatical abilities in a mainly “behavioural” fashion, by considering the model’s output. In this paper, we take it as established that neu- ral language models have indeed learned a great number of non-trivial linguistic patterns and ask instead how language models come to show this behaviour, and, more specifically, what kind of information they use to come to their decisions. There exist already a number of approaches that look inside the high-dimensional vector represen- tations and non-linear functions of these models, trying to track the flow of information. In the next section, we will review some of that work, dis- tinguishing between hypothesis-driven and data- driven methods. We highlight in particular one method called Contextual Decomposition (CD, Murdoch et al., 2018), that combines the strengths of hypothesis- and data-driven analysis methods. In …

Content truncated to 500 words