Language Modeling with Shared Grammar

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4442–4453 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4442 Language Modeling with Shared Grammar Yuyu Zhang College of Computing Georgia Institute of Technology yuyu@gatech.edu Le Song College of Computing Georgia Institute of Technology lsong@cc.gatech.edu Abstract Sequential recurrent neural networks have achieved superior performance on language modeling, but overlook the structure informa- tion in natural language. Recent works on structure-aware models have shown promis- ing results on language modeling. However, how to incorporate structure knowledge on corpus without syntactic annotations remains an open problem. In this work, we propose neural variational language model (NVLM), which enables the sharing of grammar knowl- edge among different corpora. Experimen- tal results demonstrate the effectiveness of our framework on two popular benchmark datasets. With the help of shared grammar, our language model converges significantly faster to a lower perplexity on new training corpus. 1 Introduction Language modeling has been a long-standing fun- damental task in natural language processing. In recent years, sequential recurrent neural networks (RNNs) based language models have made aston- ishing progress, which achieve remarkable results on various benchmark datasets (Mikolov et al., 2010a; Jozefowicz et al., 2016; Melis et al., 2017; Elbayad et al., 2018; Gong et al., 2018; Takase et al., 2018; Dai et al., 2019). Despite the huge success, the structure information in natural lan- guage is largely overlooked due to the structural limit of sequential RNN-based language models. Recently, researchers have explored to ex- plicitly exploit the latent structures in natu- ral language, such as recurrent neural network grammars (RNNGs; Dyer et al., 2016; Kuncoro et al., 2017) and parsing-reading-predict networks (PRPNs; Shen et al., 2017). These structure- aware models have shown promising results on language modeling, demonstrating that the latent nested structure in language indeed helps improve sequential language models. Models like RNNG exploit treebank data with syntactic annotations to learn grammar, which is then used to improve language model performance by a significant mar- gin. This is definitely intriguing, but we have to pay the cost: accurate syntactic annotation is very costly, and treebank data such as the Penn Tree- bank (Marcus et al., 1993) is typically small-scale and not open to the public for free. On new corpus which has no syntactic anno- tations, how to improve language modeling with grammar knowledge? This is an important and challenging open problem. As a motivating ex- ample, we conduct a simple experiment by train- ing a RNN language model on one corpus and testing it on another, and report the results in Ta- ble 1. The RNN language model performs terri- bly when training and testing on different datasets, which is reasonable since the data distribution may vary dramatically on different corpora. Training from scratch on every new corpus is obviously not good enough: 1) it is computationally expensive and not data-efficient; 2) the size of target cor- pus may be too small to train a decent RNN-based language …

Content truncated to 500 words