Overestimation of Syntactic Representation in Neural Language Models

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1757–1762 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1757 Overestimation of Syntactic Representation in Neural Language Models Jordan Kodner University of Pennsylvania Dept. of Linguistics jkodner@sas.upenn.edu Nitish Gupta University of Pennsylvania Dept. of Computer and Information Science nitishg@seas.upenn.edu Abstract With the advent of powerful neural language models over the last few years, research atten- tion has increasingly focused on what aspects of language they represent that make them so successful. Several testing methodologies have been developed to probe models’ syntac- tic representations. One popular method for determining a model’s ability to induce syn- tactic structure trains a model on strings gen- erated according to a template then tests the model’s ability to distinguish such strings from superficially similar ones with different syntax. We illustrate a fundamental problem with this approach by reproducing positive results from a recent paper with two non-syntactic baseline language models: an n-gram model and an LSTM model trained on scrambled inputs. 1 Introduction In recent years, RNN-based systems have proven excellent at a wide range of NLP tasks, sometimes achieving or even surpassing human performance on popular benchmarks. Their success stems from the complex but hard to interpret, representations that they learn from data. Given that syntax plays a critical role in human language competence, it is natural to ask whether part of what makes these models successful on language tasks is an ability to encode something akin to syntax. This question pertains to syntax “in the meaning- ful sense,” that is, the latent, hierarchical, largely context-free phrase structure underpinning human language as opposed to superficial or shallow is- sues of word order (Chomsky, 1957; Marcus, 1984; Everaert et al., 2015; Linzen et al., 2016). Clearly, syntactic information can be explicitly incorporated into neural systems to great effect (e.g., Dyer et al., 2016; Swayamdipta et al., 2018). Less certain is whether such systems induce something akin to hi- erarchical structure (henceforth, “syntax”) on their own when not explicitly taught to do so. Uncovering what an RNN actually represents is notoriously difficult, and several methods for prob- ing RNNs’ linguistic representations have been de- veloped to approach the problem. Most directly, one can extract finite automata (e.g., Weiss et al., 2017) from the network or measure its state as it processes inputs to determine which neurons attend to what features (e.g., Shi et al., 2016; Linzen et al., 2016; Tenney et al., 2019). Alternatively, one can present a task which only a syntactic model should be able to solve, such as grammaticality discrim- ination or an agreement task, and then infer if a model has syntactic representations based on its behavior (Linzen et al., 2016; Ettinger et al., 2018; Gulordava et al., 2018; Warstadt et al., 2019). In practice, simple sentences far outnumber the ones that require syntax in any natural corpus, which may obscure evaluation (Linzen et al., 2016). One way around this, referred to here as template- based probing, …

Content truncated to 500 words