Structural Guidance for Transformer Language Models

Structural Guidance for Transformer Language Models Peng Qian1 Tahira Naseem2 Roger Levy1 Ram´on Fernandez Astudillo2 1 Department of Brain and Cognitive Sciences, MIT 2 IBM Research pqian@mit.edu tnaseem@us.ibm.com rplevy@mit.edu ramon.astudillo@ibm.com Abstract Transformer-based language models pre- trained on large amounts of text data have proven remarkably successful in learning generic transferable linguistic representations. Here we study whether structural guidance leads to more human-like systematic linguistic generalization in Transformer language models without resorting to pre-training on very large amounts of data. We explore two general ideas. The “Generative Parsing” idea jointly models the incremental parse and word sequence as part of the same sequence modeling task. The “Structural Scaffold” idea guides the language model’s representation via additional structure loss that separately predicts the incremental constituency parse. We train the proposed models along with a vanilla Transformer language model baseline on a 14 million-token and a 46 million-token subset of the BLLIP dataset, and evaluate models’ syntactic generalization perfor- mances on SG Test Suites and sized BLiMP. Experiment results across two benchmarks suggest converging evidence that generative structural supervisions can induce more robust and humanlike linguistic generalization in Transformer language models without the need for data intensive pre-training. 1 Introduction Pre-trained Transformer architectures have led to huge progress in building more human-like lan- guage processing systems (Radford et al.; Devlin et al., 2019; Brown et al., 2020, among others). These models achieve impressive perplexity results on language modelling datasets, perform well on grammatical judgments (Warstadt et al., 2020), and provide useful linguistic representations that ben- efit a wide range of downstream tasks. Probing analyses also suggest that these models learn to im- plicitly encode syntactic information (Hewitt and Manning, 2019; Clark et al., 2019) that may sup- port better linguistic generalization than recurrent neural network architectures (RNNs). However, the Transformer architecture (Vaswani et al., 2017) is an interesting subject of study be- yond its success in transfer-learning settings. Trans- former models lack the inductive biases of RNNs. Rather than maintaining vector-valued state and updating it in a recurrent manner, auto-regressive Transformer models encode all past decisions si- multaneously at each inference step, thanks to a self-attention mechanism. The only notion of se- quence order is also given by position embeddings summed to content embeddings in both input and auto-regressive signals. Previous works have shown the advantage of structural supervision in RNNs in learning to main- tain syntactic states and non-local dependencies (Kuncoro et al., 2018; Wilcox et al., 2019; Futrell et al., 2019). It remains an open question whether Transformer language models can similarly benefit from generative structural supervision, and what form of structural supervision would more effec- tively induce human-like syntactic generalization. This work hypothesizes that the Transformer lan- guage model may benefit from explicit generative structural supervision to systematically generalize syntactic knowledge. Here we explore two ma- jor classes of structural guidance for Transformer language models based on joint modeling of lan- guage and constituency parses. The “generative parsing as language modeling” approach builds a Transformer-parameterized model to learn to pre- …

Content truncated to 500 words