Grammar Induction with Neural Language Models: An Unusual Replication

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 371–373 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 371 Grammar Induction with Neural Language Models: An Unusual Replication Phu Mon Htut1 pmh330@nyu.edu Kyunghyun Cho1,2 kyunghyun.cho@nyu.edu Samuel R. Bowman1,2,3 bowman@nyu.edu 1Center for Data Science New York University 60 Fifth Avenue New York, NY 10011 2Dept. of Computer Science New York University 60 Fifth Avenue New York, NY 10011 3Dept. of Linguistics New York University 10 Washington Place New York, NY 10003 1 Introduction Grammar induction is the task of learning syntac- tic structure without the expert-labeled treebanks (Charniak and Carroll, 1992; Klein and Manning, 2002). Recent work on latent tree learning of- fers a new family of approaches to this problem by inducing syntactic structure using the supervision from a downstream NLP task (Yogatama et al., 2017; Maillard et al., 2017; Choi et al., 2018). In a recent paper published at ICLR, Shen et al. (2018) introduce such a model and report near state-of- the-art results on the target task of language mod- eling, and the first strong latent tree learning re- sult on constituency parsing. During the analy- sis of this model, we discover issues that make the original results hard to trust, including tuning and even training on what is effectively the test set. Here, we analyze the model under different configurations to understand what it learns and to identify the conditions under which it succeeds. We find that this model represents the first empiri- cal success for neural network latent tree learning, and that neural language modeling warrants fur- ther study as a setting for grammar induction. 2 WeBackground and Experiments analyze the Parsing-Reading-Predict- Network (PRPN; Shen et al., 2018), which uses convolutional networks with a form of structured attention (Kim et al., 2017) rather than recursive neural networks (Goller and Kuchler, 1996; Socher et al., 2011) to learn trees while perform- ing straightforward backpropagation training on a language modeling objective. The structure of the model seems rather suboptimal: Since the parser is trained as part of a language model, it makes parsing greedily, with no access to any words to the right of the point where each parsing decision must be made. The experiments on language modeling and A crusade of NO to the consumption of drugs is imperative . Figure 1: Parses from PRPN-LM trained on AllNLI. parsing are carried out using different configura- tions of the model—PRPN-LM tuned for language modeling, and PRPN-UP for (unsupervised) pars- ing. PRPN-LM is much larger than PRPN-UP, with embedding layer that is 4 times larger and the number of units per layer that is 3 times larger. In the PRPN-UP experiments, we observe that the WSJ data is not split, such that the test data is used without parse information for training. This im- plies that the parsing results of PRPN-UP may not be generalizable in the way usually expected of machine learning evaluation results. We train PRPN on sentences from …

Content truncated to 500 words