Scalable Syntax-Aware Language Models Using Knowledge Distillation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3472–3484 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3472 Scalable Syntax-Aware Language Models Using Knowledge Distillation Adhiguna Kuncoro♠♦ Chris Dyer♠ Laura Rimell♠ Stephen Clark♠ Phil Blunsom♠♦ ♠DeepMind, London, UK ♦Department of Computer Science, University of Oxford, UK {akuncoro,cdyer,laurarimell,clarkstephen,pblunsom}@google.com Abstract Prior work has shown that, on small amounts of training data, syntactic neural language models learn structurally sensitive generalisa- tions more successfully than sequential lan- guage models. However, their computational complexity renders scaling difficult, and it re- mains an open question whether structural bi- ases are still necessary when sequential mod- els have access to ever larger amounts of train- ing data. To answer this question, we intro- duce an efficient knowledge distillation (KD) technique that transfers knowledge from a syn- tactic language model trained on a small cor- pus to an LSTM language model, hence en- abling the LSTM to develop a more struc- turally sensitive representation of the larger training data it learns from. On targeted syn- tactic evaluations, we find that, while sequen- tial LSTMs perform much better than previ- ously reported, our proposed technique sub- stantially improves on this baseline, yielding a new state of the art. Our findings and anal- ysis affirm the importance of structural biases, even in models that learn from large amounts of data. 1 Introduction Language models (LMs) based on sequential LSTMs (Hochreiter and Schmidhuber, 1997) have numerous practical applications, but it has also been shown that they do not always develop accu- rate syntactic generalisations (Marvin and Linzen, 2018). Thus, one strategy for improving LSTMs is to change their biases to facilitate more linguis- tically valid generalisations. This paper introduces a scalable method for in- troducing syntactic biases to LSTMs (and indeed, to any left-to-right language model trained with a cross-entropy objective) by distilling knowl- edge (Bucilˇa et al., 2006; Hinton et al., 2015) from recurrent neural network grammars (Dyer et al., 2016, RNNGs). RNNGs have been shown to suc- cessfully capture non-local syntactic dependencies (Kuncoro et al., 2018), achieve excellent parsing performance (Kuncoro et al., 2017; Fried et al., 2017), and correlate well with encephalography signals (Hale et al., 2018). Unfortunately, these benefits come at the expense of scalability, since the hierarchical constituent composition process (§3) within RNNGs means that the structure of the computation graph for a sentence varies according to its tree structure. Even with the help of auto- matic dynamic batching (Neubig et al., 2017a,b), RNNGs can be ten times slower to train than a comparable LSTM as they benefit less from spe- cialised hardware like GPUs. As such, RNNGs are an impractical alternative to computationally convenient architectures that are used to build lan- guage models from massive corpora (Peters et al., 2018; Howard and Ruder, 2018; Devlin et al., 2019; Radford et al., 2019). As RNNGs are hard to scale, we instead use the predictions of an RNNG teacher model trained on a small training set, to …

Content truncated to 500 words