Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 240–248 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 240 Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information Mario Giulianelli University of Amsterdam Jack Harding University of Amsterdam {mario.giulianelli, jack.harding, florian.mohnert}@student.uva.nl Florian Mohnert University of Amsterdam Dieuwke Hupkes ILLC, University of Amsterdam d.hupkes@uva.nl Willem Zuidema ILLC, University of Amsterdam w.h.zuidema@uva.nl Abstract How do neural language models keep track of number agreement between subject and verb? We show that ‘diagnostic classifiers’, trained to predict number from the internal states of a language model, provide a detailed understanding of how, when, and where this information is represented. Moreover, they give us insight into when and where number information is corrupted in cases where the language model ends up making agreement errors. To demonstrate the causal role played by the representations we find, we then use agreement information to influence the course of the LSTM during the processing of difficult sentences. Results from such an intervention reveal a large increase in the language model’s accuracy. Together, these results show that diagnostic classifiers give us an unrivalled detailed look into the representation of linguistic information in neural models, and demonstrate that this knowledge can be used to improve their performance. 1 Introduction Machine learning models for estimating the prob- abilities of potential next words (and hence, for predicting the next word) in a running text have seen enormous improvements in performance over the last few years (Merity et al., 2018). These newer models—all based on deep learning tech- niques such as LSTMs (Hochreiter and Schmid- huber, 1997)—allow some language technologies, such as speech recognisers, to reach ‘human par- ity’. From their high accuracy and from further analysis, it is clear that LSTM-based language models have learned a great deal about both short and long distance relations in sentences and dis- course. In particular, Gulordava et al. (2018) re- port that for several languages, their LSTM-based language model performs remarkably well on a set of long-distance number agreement tasks. The Gulordava study, however, does not clarify which components of the LSTM are responsible for storing or processing syntactic features, and how such features are represented. Understanding how trained recurrent networks such as LSTMs might represent syntax and other structural infor- mation is currently a key area of research. Popular approaches include visualising the state space of these networks, performing ablations to the net- work, or using the internal states of the networks for some auxiliary task (e.g., Adi et al., 2016; K´ad´ar et al., 2017; Conneau et al., 2018; Khan- delwal et al., 2018). In this paper, we analyse the phenomenon of subject-verb agreement in English using the di- agnostic classification approach of Hupkes et al. (2018). We start with replicating the results of Gu- lordava et al. (2018) on English, and we then show that diagnostic classifiers can be used to give a fine-grained analysis of how neural language …

Content truncated to 500 words