Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale

Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale Laurent Sartran1 Samuel Barrett2 Adhiguna Kuncoro1,2 Miloˇs Stanojevi´c1 Phil Blunsom2 Chris Dyer1 1DeepMind, UK 2University of Oxford, UK {lsartran,akuncoro,stanojevic,cdyer}@deepmind.com samuelbarrett1234@btinternet.com, phil.blunsom@cs.ox.ac.uk Abstract We introduce Transformer Grammars (TGs), a novel class of Transformer language models that combine (i) the expressive power, scala- bility, and strong performance of Transform- ers and (ii) recursive syntactic compositions, which here are implemented through a spe- cial attention mask and deterministic transfor- mation of the linearized tree. We find that TGs outperform various strong baselines on sentence-level language modeling perplex- ity, as well as on multiple syntax-sensitive language modeling evaluation metrics. Addi- tionally, we find that the recursive syntactic composition bottleneck which represents each sentence as a single vector harms perplexity on document-level language modeling, pro- viding evidence that a different kind of mem- ory mechanism—one that is independent of composed syntactic representations—plays an important role in current successful models of long text. 1 Introduction Transformer language models (LMs) that are trained on vast amounts of data have achieved remarkable success at various NLP benchmarks (Peters et al., 2018; Devlin et al., 2019; Brown et al., 2020, inter alia). Intriguingly, this success is achieved by models that lack an explicit mod- eling of hierarchical syntactic structures, which were hypothesized by decades of linguistic re- search to be necessary for good generalization (Chomsky, 1957; Everaert et al., 2015). This nat- urally leaves a question: To what extent can we further improve the performance of Transformer LMs, through an inductive bias that encourages the model to explain the data by the means of recursive syntactic compositions? Although the benefits of modeling recursive syntax have been shown at the small data and model scales, such as in the case of recurrent neural network gram- mars (Dyer et al., 2016; Futrell et al., 2019; Kim et al., 2019; Hu et al., 2020; Noji and Oseki, 2021, inter alia), it remains an open question whether—and to what extent—a similar design principle is still beneficial for Transformer LMs at larger scales. In this paper, we aim to answer these questions by introducing Transformer Gram- mars (TGs)—a novel class of Transformer lan- guage models that combine: (i) the expressive power, scalability, and strong performance of Transformer-XL (Dai et al., 2019); (ii) joint mod- eling of surface strings x and their correspond- ing phrase-structure trees y, namely, p(x, y); and (iii) an inductive bias that constrains the model to explain the data through built-in recursive syntactic composition operations. By implement- ing these recursive compositions through a novel modification of the Transformer-XL attention mask, TGs retain the computational efficiency of standard Transformer-XLs, enabling them to avoid the limitations of LSTM-based recurrent neural network grammars (Dyer et al., 2016, RNNGs), which have been proven difficult to scale (Kuncoro et al., 2019; Noji and Oseki, 2021). TGs are related to the recent work of Qian et al. (2021) that similarly aims to augment gener- ative Transformer language models with a stronger modeling of syntactic structures, …

Content truncated to 500 words