Structural Supervision Improves Learning of Non-Local Grammatical Dependencies

Proceedings of NAACL-HLT 2019, pages 3302–3312 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3302 Structural Supervision Improves Learning of Non-Local Grammatical Dependencies Ethan Wilcox1, Peng Qian2, Richard Futrell3, Miguel Ballesteros4, and Roger Levy2 1Department of Linguistics, Harvard University, wilcoxeg@g.harvard.edu 2Department of Brain and Cognitive Sciences, MIT, {pqian,rplevy}@mit.edu 3Department of Language Science, UC Irvine, rfutrell@uci.edu 4IBM Research, MIT-IBM Watson AI Lab miguel.ballesteros@ibm.com Abstract State-of-the-art LSTM language models trained on large corpora learn sequential contingencies in impressive detail and have been shown to acquire a number of non-local grammatical dependencies with some success. Here we investigate whether supervision with hierarchical structure enhances learning of a range of grammatical dependencies, a ques- tion that has previously been addressed only for subject-verb agreement. Using controlled experimental methods from psycholinguistics, we compare the performance of word-based LSTM models versus two models that rep- resent hierarchical structure and deploy it in left-to-right processing: Recurrent Neural Network Grammars (RNNGs) (Dyer et al., 2016) and a incrementalized version of the Parsing-as-Language-Modeling configuration from Charniak et al. (2016). Models are tested on a diverse range of configurations for two classes of non-local grammatical dependen- cies in English—Negative Polarity licensing and Filler–Gap Dependencies. Using the same training data across models, we find that structurally-supervised models outperform the LSTM, with the RNNG demonstrating best results on both types of grammatical dependencies and even learning many of the Island Constraints on the filler–gap depen- dency. Structural supervision thus provides data efficiency advantages over purely string- based training of neural language models in acquiring human-like generalizations about non-local grammatical dependencies. 1 Introduction Long Short-Term Memory Recurrent Neural Net- works (LSTMs) (Hochreiter and Schmidhuber, 1997) have achieved state of the art language mod- eling performance (Jozefowicz et al., 2016) and have been shown to indirectly learn a number of non-local grammatical dependencies, such as subject-verb number agreement and filler-gap li- censing (Linzen et al., 2016; Wilcox et al., 2018), although they fail to learn others, such as Nega- tive Polarity Item and anaphoric pronoun licensing (Marvin and Linzen, 2018; Futrell et al., 2018). LSTMs, however, require large amounts of train- ing data and remain relatively uninterpretable. One model that attempts to address both these is- sues is the Recurrent Neural Network Grammar (Dyer et al., 2016). RNNGs are generative mod- els, which represent hierarchical syntactic struc- ture and use neural control to deploy it in left-to- right processing. They can achieve state-of-the-art broad-coverage scores on language modeling and phrase structure parsing tasks, learn Noun Phrase headedness (Kuncoro et al., 2016), and outper- form linear models at learning subject-verb num- ber agreement (Kuncoro et al., 2018). In this work, we comparatively evaluate LSTMs, RNNGs and a third model trained using syntactic supervision—similar to the Parsing-as- Language-Modeling configuration from Charniak et al. (2016)—by conducting side-by-side tests on two novel English grammatical dependencies, de- ploying methodology from psycholinguistics. In this paradigm, the language models are fed with hand-crafted sentences, designed to draw out be- havior that belies whether they have learned …

Content truncated to 500 words