Syntactic Structure Distillation Pretraining for Bidirectional Encoders

Syntactic Structure Distillation Pretraining for Bidirectional Encoders Adhiguna Kuncoro∗♠♦ Lingpeng Kong∗♠ Daniel Fried∗♣ Dani Yogatama♠ Laura Rimell♠ Chris Dyer♠ Phil Blunsom♠♦ ♠DeepMind, London, UK ♦Department of Computer Science, University of Oxford, UK ♣Computer Science Division, University of California, Berkeley, CA, USA {akuncoro,lingpenk,dyogatama,laurarimell,cdyer,pblunsom}@google.com dfried@cs.berkeley.edu Abstract Textual representation learners trained on large amounts of data have achieved notable success on downstream tasks; intriguingly, they have also performed well on challenging tests of syntactic competence. Hence, it remains an open question whether scalable learners like BERT can become fully proficient in the syntax of natural language by virtue of data scale alone, or whether they still benefit from more explicit syntactic biases. To answer this question, we introduce a knowledge distillation strategy for injecting syntactic biases into BERT pretraining, by distilling the syntactically informative predictions of a hierarchical—albeit harder to scale—syntactic language model. Since BERT models masked words in bidirectional context, we propose to distill the approximate marginal distribution over words in context from the syntactic LM. Our approach reduces relative error by 2–21% on a diverse set of structured prediction tasks, although we obtain mixed results on the GLUE benchmark. Our findings demonstrate the benefits of syntactic biases, even for representation learners that exploit large amounts of data, and contribute to a better understanding of where syntactic biases are helpful in benchmarks of natural language understanding. 1 Introduction Large-scale textual representation learners trained with variants of the language modeling (LM) obj- ective have achieved remarkable success on down- stream tasks (Peters et al., 2018; Devlin et al., 2019; Yang et al., 2019). Furthermore, these mo- ∗Equal contribution. dels have also been shown to perform remark- ably well at syntactic grammaticality judgment tasks (Goldberg, 2019), and encode substantial amounts of syntax in their learned representa- tions (Liu et al., 2019a; Tenney et al., 2019a,b; Hewitt and Manning, 2019; Jawahar et al., 2019). Intriguingly, success on these syntactic tasks has been achieved by Transformer architectures (Vaswani et al., 2017) that lack explicit notions of hierarchical syntactic structures. Based on such evidence, it would be tempting to conclude that data scale alone is all we need to learn the syntax of natural language. Nevertheless, recent findings that systematically compare the syntactic competence of models trained at varying data scales suggest that model inductive biases are in fact more important than data scale for acquiring syntactic competence (Hu et al., 2020). Two natural questions, therefore, are the following: Can representation learners that work well at scale still benefit from explicit syntactic biases? And where exactly would such syntactic biases be helpful in different language understanding tasks? Here we work towards answering these questions by devising a new pretraining strategy that injects syntactic biases into a BERT (Devlin et al., 2019) learner that works well at scale. We hypothesize that this approach can improve the competence of BERT on various tasks, which provides evidence for the benefits of syntactic biases in large-scale models. Our approach is based on the prior work of Kuncoro et al. (2019), who …

Content truncated to 500 words