What Do Recurrent Neural Network Grammars Learn About Syntax?

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1249–1258, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics What Do Recurrent Neural Network Grammars Learn About Syntax? Adhiguna Kuncoro♠ Miguel Ballesteros♦ Lingpeng Kong♠ Chris Dyer♠♣ Graham Neubig♠ Noah A. Smith♥ ♠School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA ♦IBM T.J. Watson Research Center, Yorktown Heights, NY, USA ♣DeepMind, London, UK ♥Computer Science & Engineering, University of Washington, Seattle, WA, USA {akuncoro,lingpenk,gneubig}@cs.cmu.edu miguel.ballesteros@ibm.com, cdyer@google.com, nasmith@cs.washington.edu Abstract Recurrent neural network grammars (RNNG) are a recently proposed prob- ablistic generative modeling family for natural language. They show state-of- the-art language modeling and parsing performance. We investigate what in- formation they learn, from a linguistic perspective, through various ablations to the model and the data, and by aug- menting the model with an attention mechanism (GA-RNNG) to enable closer inspection. We find that explicit modeling of composition is crucial for achieving the best performance. Through the attention mechanism, we find that headedness plays a central role in phrasal represen- tation (with the model’s latent attention largely agreeing with predictions made by hand-crafted head rules, albeit with some important differences). By training grammars without nonterminal labels, we find that phrasal representations depend minimally on nonterminals, providing support for the endocentricity hypothesis. 1 Introduction In this paper, we focus on a recently proposed class of probability distributions, recurrent neural network grammars (RNNGs; Dyer et al., 2016), designed to model syntactic derivations of sen- tences. We focus on RNNGs as generative proba- bilistic models over trees, as summarized in §2. Fitting a probabilistic model to data has often been understood as a way to test or confirm some aspect of a theory. We talk about a model’s as- sumptions and sometimes explore its parameters or posteriors over its latent variables in order to gain understanding of what it “discovers” from the data. In some sense, such models can be thought of as mini-scientists. Neural networks, including RNNGs, are capa- ble of representing larger classes of hypotheses than traditional probabilistic models, giving them more freedom to explore. Unfortunately, they tend to be bad mini-scientists, because their parameters are difficult for human scientists to interpret. RNNGs are striking because they obtain state- of-the-art parsing and language modeling perfor- mance. Their relative lack of independence as- sumptions, while still incorporating a degree of linguistically-motivated prior knowledge, affords the model considerable freedom to derive its own insights about syntax. If they are mini-scientists, the discoveries they make should be of particular interest as propositions about syntax (at least for the particular genre and dialect of the data). This paper manipulates the inductive bias of RNNGs to test linguistic hypotheses.1 We be- gin with an ablation study to discover the impor- tance of the composition function in §3. Based on the findings, we augment the RNNG composi- tion function with a novel gated attention mech- anism (leading to the GA-RNNG) to incorporate more interpretability into the model …

Content truncated to 500 words