Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6829–6839, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6829 Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models Isabel Papadimitriou Stanford University isabelvp@stanford.edu Dan Jurafsky Stanford University jurafsky@stanford.edu Abstract We propose transfer learning as a method for analyzing the encoding of grammatical struc- ture in neural language models. We train LSTMs on non-linguistic data and evaluate their performance on natural language to as- sess which kinds of data induce generalizable structural features that LSTMs can use for nat- ural language. We find that training on non- linguistic data with latent structure (MIDI mu- sic or Java code) improves test performance on natural language, despite no overlap in surface form or vocabulary. To pinpoint the kinds of abstract structure that models may be encod- ing to lead to this improvement, we run simi- lar experiments with two artificial parentheses languages: one which has a hierarchical recur- sive structure, and a control which has paired tokens but no recursion. Surprisingly, training a model on either of these artificial languages leads the same substantial gains when testing on natural language. Further experiments on transfer between natural languages controlling for vocabulary overlap show that zero-shot per- formance on a test language is highly corre- lated with typological syntactic similarity to the training language, suggesting that represen- tations induced by pre-training correspond to the cross-linguistic syntactic properties. Our results provide insights into the ways that neu- ral models represent abstract syntactic struc- ture, and also about the kind of structural in- ductive biases which allow for natural lan- guage acquisition. 1 1 Introduction Understanding how neural language models learn and represent syntactic structure is an important an- alytic question for NLP. Recent work has directly probed the internal activations of models (Conneau 1We release code to construct the corpora and run our experiments at https://github.com/toizzy/ tilt-transfer Random Uniform Random Zipf Music Code Nesting Parens Flat Parens Japanese English Portuguese Pretraining Language (L1) 50 100 150 200 250 300 350 400 450 500 Ppl when tested on Spanish (L2) (Lower is better) Figure 1: We find that LSTM LMs can utilize vari- ous types of non-linguistic structure to help learn to model human language, and that nested hierarchical structure does not lead to more expressive encodings than flat, head-dependency pair structure. We also find that LSTM LMs learn representations that corre- late with typological syntactic feature distance, allow- ing them to transfer more effectively from languages which are grammatically similar. et al., 2018a; Dalvi et al., 2019; Hewitt and Man- ning, 2019; Clark et al., 2019), or fed them curated inputs that depend on complex syntax (Linzen et al., 2016; Gulordava et al., 2018; Talmor et al., 2019; McCoy et al., 2020), in order to uncover latent syntactic awareness. We propose a different approach: we measure the structural awareness of a language model by studying how much this structure acts as an induc- tive bias to improve …

Content truncated to 500 words