Linguistic Knowledge and Transferability of Contextual Representations

Proceedings of NAACL-HLT 2019, pages 1073–1094 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1073 Linguistic Knowledge and Transferability of Contextual Representations Nelson F. Liu♠♥∗ Matt Gardner♣ Yonatan Belinkov♦ Matthew E. Peters♣ Noah A. Smith♠♣ ♠Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA, USA ♥Department of Linguistics, University of Washington, Seattle, WA, USA ♣Allen Institute for Artificial Intelligence, Seattle, WA, USA ♦Harvard John A. Paulson School of Engineering and Applied Sciences and MIT Computer Science and Artificial Intelligence Laboratory, Cambridge, MA, USA {nfliu,nasmith}@cs.washington.edu {mattg,matthewp}@allenai.org, belinkov@seas.harvard.edu Abstract Contextual word representations derived from large-scale neural language models are suc- cessful across a diverse set of NLP tasks, suggesting that they encode useful and trans- ferable features of language. To shed light on the linguistic knowledge they capture, we study the representations produced by sev- eral recent pretrained contextualizers (variants of ELMo, the OpenAI transformer language model, and BERT) with a suite of sixteen di- verse probing tasks. We find that linear mod- els trained on top of frozen contextual repre- sentations are competitive with state-of-the-art task-specific models in many cases, but fail on tasks requiring fine-grained linguistic knowl- edge (e.g., conjunct identification). To inves- tigate the transferability of contextual word representations, we quantify differences in the transferability of individual layers within con- textualizers, especially between recurrent neu- ral networks (RNNs) and transformers. For in- stance, higher layers of RNNs are more task- specific, while transformer layers do not ex- hibit the same monotonic trend. In addition, to better understand what makes contextual word representations transferable, we compare lan- guage model pretraining with eleven super- vised pretraining tasks. For any given task, pretraining on a closely related task yields bet- ter performance than language model pretrain- ing (which is better on average) when the pre- training dataset is fixed. However, language model pretraining on more data gives the best results. 1 Introduction Pretrained word representations (Mikolov et al., 2013; Pennington et al., 2014) are a key compo- nent of state-of-the-art neural NLP models. Tra- ditionally, these word vectors are static—a single *Work done while at the Allen Institute for Artificial In- telligence. Figure 1: An illustration of the probing model setup used to study the linguistic knowledge within contex- tual word representations. vector is assigned to each word. Recent work has explored contextual word representations (hence- forth: CWRs), which assign each word a vector that is a function of the entire input sequence; this enables them to model the use of words in context. CWRs are typically the outputs of a neural net- work (which we call a contextualizer) trained on tasks with large datasets, such as machine trans- lation (McCann et al., 2017) and language mod- eling (Peters et al., 2018a). CWRs are extraordi- narily effective—using them in place of traditional static word vectors within the latest models leads to large gains across a variety of NLP tasks. The broad success of CWRs indicates that they encode useful, transferable features of …

Content truncated to 500 words