Influence Paths for Characterizing Subject-Verb Number Agreement in LSTM Language Models

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4748–4757 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4748 Influence Paths for Characterizing Subject-Verb Number Agreement in LSTM Language Models Kaiji Lu Carnegie Mellon University kaijil@andrew.cmu.edu Piotr Mardziel Carnegie Mellon University piotrm@gmail.com Klas Leino Carnegie Mellon University kleino@cs.cmu.edu Matt Fedrikson Carnegie Mellon University mfredrik@cmu.edu Anupam Datta Carnegie Mellon University danupam@cmu.edu Abstract LSTM-based recurrent neural networks are the state-of-the-art for many natural language pro- cessing (NLP) tasks. Despite their perfor- mance, it is unclear whether, or how, LSTMs learn structural features of natural languages such as subject-verb number agreement in En- glish. Lacking this understanding, the general- ity of LSTMs on this task and their suitabil- ity for related tasks remains uncertain. Fur- ther, errors cannot be properly attributed to a lack of structural capability, training data omissions, or other exceptional faults. We in- troduce influence paths, a causal account of structural properties as carried by paths across gates and neurons of a recurrent neural net- work. The approach refines the notion of in- fluence (the subject’s grammatical number has influence on the grammatical number of the subsequent verb) into a set of gate-level or neuron-level paths. The set localizes and seg- ments the concept (e.g., subject-verb agree- ment), its constituent elements (e.g., the sub- ject), and related or interfering elements (e.g., attractors). We exemplify the methodology on a widely-studied multi-layer LSTM lan- guage model, demonstrating its accounting for subject-verb number agreement. The results offer both a finer and a more complete view of an LSTM’s handling of this structural as- pect of the English language than prior results based on diagnostic classifiers and ablation. 1 Introduction Traditional rule-based NLP techniques can cap- ture syntactic structures, while statistical NLP tech- niques, such as n-gram models, can heuristically integrate semantics of a natural language. Mod- ern RNN-based models such as Long Short-Term Memory (LSTM) models are tasked with incorpo- rating both semantic features from the statistical associations in their training corpus, and structural features generalized from the same. In: Out: boys behind the tree (run) The s0 s1 s2 s3 s4 Cell c0 1 Candidate Cell ˜c1 1 Cell c1 1 Candidate Cell ˜c0 1 Hidden h0 1 c1 2 c1 3 c1 4 h1 4 c0 4 ˜c1 4 ˜c0 4 h0 4 agreement s4(run) - s4(runs) grammatical number boys − boy+boys 2 Figure 1: Subject-verb agreement task for a 2-layer LSTM language model, and primary paths across var- ious LSTM gates implementing subject-verb number agreement. A language model assigns score s to each word. Agreement is the score of the correctly num- bered verb minus that of the incorrectly numbered verb. Despite evidence that LSTMs can capture syntac- tic rules in artificial languages (Gers and Schmid- huber, 2001), it is unclear whether they are as ca- pable in natural languages (Linzen et al., 2016; Lakretz et al., 2019) in the context of rules such as subject-verb number agreement, especially when not supervised for the …

Content truncated to 500 words