Unsupervised Recurrent Neural Network Grammars

Proceedings of NAACL-HLT 2019, pages 1105–1117 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1105 Unsupervised Recurrent Neural Network Grammars Yoon Kim† Alexander M. Rush† Lei Yu3 Adhiguna Kuncoro‡,3 Chris Dyer3 G´abor Melis3 †Harvard University ‡University of Oxford 3DeepMind {yoonkim,srush}@seas.harvard.edu {leiyu,akuncoro,cdyer,melisgl}@google.com Abstract Recurrent neural network grammars (RNNG) are generative models of language which jointly model syntax and surface structure by incrementally generating a syntax tree and sentence in a top-down, left-to-right order. Supervised RNNGs achieve strong language modeling and parsing performance, but re- quire an annotated corpus of parse trees. In this work, we experiment with unsupervised learning of RNNGs. Since directly marginal- izing over the space of latent trees is in- tractable, we instead apply amortized varia- tional inference. To maximize the evidence lower bound, we develop an inference net- work parameterized as a neural CRF con- stituency parser. On language modeling, unsu- pervised RNNGs perform as well their super- vised counterparts on benchmarks in English and Chinese. On constituency grammar in- duction, they are competitive with recent neu- ral language models that induce tree structures from words through attention mechanisms. 1 Introduction Recurrent neural network grammars (RNNGs) (Dyer et al., 2016) model sentences by first gen- erating a nested, hierarchical syntactic structure which is used to construct a context representation to be conditioned upon for upcoming words. Su- pervised RNNGs have been shown to outperform standard sequential language models, achieve ex- cellent results on parsing (Dyer et al., 2016; Kun- coro et al., 2017), better encode syntactic proper- ties of language (Kuncoro et al., 2018), and cor- relate with electrophysiological responses in the human brain (Hale et al., 2018). However, these all require annotated syntactic trees for training. In this work, we explore unsupervised learning of recurrent neural network grammars for language modeling and grammar induction. Work done while the first author was an intern at DeepMind. Code available at https://github.com/harvardnlp/urnng The standard setup for unsupervised structure learning is to define a generative model pθ(x, z) over observed data x (e.g. sentence) and unob- served structure z (e.g. parse tree, part-of-speech sequence), and maximize the log marginal like- lihood log pθ(x) = log � z pθ(x, z). Success- ful approaches to unsupervised parsing have made strong conditional independence assumptions (e.g. context-freeness) and employed auxiliary objec- tives (Klein and Manning, 2002) or priors (John- son et al., 2007). These strategies imbue the learn- ing process with inductive biases that guide the model to discover meaningful structures while al- lowing tractable algorithms for marginalization; however, they come at the expense of language modeling performance, particularly compared to sequential neural models that make no indepen- dence assumptions. Like RNN language models, RNNGs make no independence assumptions. Instead they encode structural bias through operations that compose linguistic constituents. The lack of independence assumptions contributes to the strong language modeling performance of RNNGs, but make unsu- pervised learning challenging. First, marginaliza- tion is intractable. Second, the biases imposed by the RNNG are relatively weak compared to those imposed …

Content truncated to 500 words