Structural Supervision Improves Few-Shot Learning and Syntactic Generalization in Neural Language Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4640–4652, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4640 Structural Supervision Improves Few-Shot Learning and Syntactic Generalization in Neural Language Models Ethan Wilcox1, Peng Qian2, Richard Futrell3, Ryosuke Kohita4, Roger Levy2 and Miguel Ballesteros5 1Department of Linguistics, Harvard University 2Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology 3Department of Language Science, UC Irvine 4IBM Research 5Amazon AI wilcoxeg@g.harvard.edu, {pqian,rplevy}@mit.edu rfutrell@uci.edu, kohi@ibm.com, ballemig@amazon.com Abstract Humans can learn structural properties about a word from minimal experience, and de- ploy their learned syntactic representations uniformly in different grammatical contexts. We assess the ability of modern neural lan- guage models to reproduce this behavior in En- glish and evaluate the effect of structural super- vision on learning outcomes. First, we assess few-shot learning capabilities by developing controlled experiments that probe models’ syn- tactic nominal number and verbal argument structure generalizations for tokens seen as few as two times during training. Second, we assess invariance properties of learned repre- sentation: the ability of a model to transfer syn- tactic generalizations from a base context (e.g., a simple declarative active-voice sentence) to a transformed context (e.g., an interrogative sen- tence). We test four models trained on the same dataset: an n-gram baseline, an LSTM, and two LSTM-variants trained with explicit structural supervision (Dyer et al., 2016; Char- niak et al., 2016). We find that in most cases, the neural models are able to induce the proper syntactic generalizations after minimal expo- sure, often from just two examples during training, and that the two structurally super- vised models generalize more accurately than the LSTM model. All neural models are able to leverage information learned in base con- texts to drive expectations in transformed con- texts, indicating that they have learned some invariance properties of syntax. 1 Introduction Recurrent Neural Network language models (El- man, 1990; Hochreiter and Schmidhuber, 1997) have been shown to learn many aspects of natu- ral language syntax including a number of long- Miguel conducted this work while at IBM Research Scripts and data for this paper can be found online at https://github.com/wilcoxeg/fsl invar distance dependencies and representations of incre- mental syntactic state (Marvin and Linzen, 2018; Wilcox et al., 2018; Futrell et al., 2018). However, previous studies have not investigated the relation- ship between a token’s frequency in the training corpus and syntactic properties models learn about it. In this work, we assess neural models’ ability to make robust syntactic generalizations about a to- ken’s nominal number or verbal argument structure based on minimal exposure with the token during training. Because of the Zipfian distribution of words in a corpus, the vast majority of word types will be seen only a handful of times during training (Zipf, 1949). Therefore, the few-shot learning capa- bilities of neural LMs are critical to their robustness as an NLP system and as a cognitive model. However, human learning goes beyond simply learning syntactic properties in particular construc- tions. People apply the …

Content truncated to 500 words