Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models

Findings of the Association for Computational Linguistics: ACL 2022, pages 1352 - 1368 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models Aaron MuellerC Robert Frank7 Tal LinzenZ Luheng WangZ Sebastian SchusterZ CJohns Hopkins University 7Yale University ZNew York University amueller@jhu.edu, schuster@nyu.edu Abstract Relations between words are governed by hi- erarchical structure rather than linear ordering. Sequence-to-sequence (seq2seq) models, de- spite their success in downstream NLP appli- cations, often fail to generalize in a hierarchy- sensitive manner when performing syntactic transformations—for example, transforming declarative sentences into questions. However, syntactic evaluations of seq2seq models have only observed models that were not pre-trained on natural language data before being trained to perform syntactic transformations, in spite of the fact that pre-training has been found to induce hierarchical linguistic generalizations in language models; in other words, the syn- tactic capabilities of seq2seq models may have been greatly understated. We address this gap using the pre-trained seq2seq models T5 and BART, as well as their multilingual variants mT5 and mBART. We evaluate whether they generalize hierarchically on two transforma- tions in two languages: question formation and passivization in English and German. We find that pre-trained seq2seq models general- ize hierarchically when performing syntactic transformations, whereas models trained from scratch on syntactic transformations do not. This result presents evidence for the learnabil- ity of hierarchical syntactic information from non-annotated natural language text while also demonstrating that seq2seq models are capable of syntactic generalization, though only after exposure to much more language data than hu- man learners receive. 1 Introduction Human language is structured hierarchically. In NLP tasks like natural language inference, syn- tactic competence is a prerequisite for robust gen- eralization (e.g., McCoy et al., 2019). Probing studies have found that masked language models (MLMs) contain hierarchical representations (Ten- ney et al., 2019; Hewitt and Manning, 2019; Clark Figure 1: The poverty of the stimulus experimental de- sign. We fine-tune pre-trained seq2seq models and train small seq2seq models from scratch to perform syntac- tic transformations. The training set contains ambigu- ous examples consistent with hierarchical and linear transformation rules. The generalization set contains examples where only the hierarchical rule results in the correct output. Pre-trained models generalize using the hierarchical rule, while models trained from scratch gen- eralize using the linear rule. et al., 2019), while behavioral studies of recurrent neural language models (Linzen et al., 2016; Mar- vin and Linzen, 2018; Wilcox et al., 2018; van Schi- jndel et al., 2019) and MLMs (Goldberg, 2019; Hu et al., 2020) have found that models are largely able to capture long-range syntactic dependencies that require hierarchical representations of sentences. Recent evidence suggests that MLMs like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) can learn to make hierarchical linguistic gen- eralizations through exposure to text (Warstadt and Bowman, 2020), though acquiring many of these linguistic generalizations requires large amounts of data (Warstadt et al., 2020). However, this evidence comes from binary …

Content truncated to 500 words