Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4465–4476 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4465 Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling Alex Wang,∗1 Jan Hula,2 Patrick Xia,2 Raghavendra Pappagari,2 R. Thomas McCoy,2, Roma Patel,3 Najoung Kim,2 Ian Tenney,4 Yinghui Huang,6 Katherin Yu,5 Shuning Jin,7 Berlin Chen8 Benjamin Van Durme,2 Edouard Grave,5 Ellie Pavlick,3,4 and Samuel R. Bowman1 1New York University, 2Johns Hopkins University, 3Brown University, 4Google AI Language, 5Facebook, 6IBM, 7University of Minnesota Duluth, 8Swarthmore College Abstract Natural language understanding has recently seen a surge of progress with the use of sentence encoders like ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2019) which are pretrained on variants of language mod- eling. We conduct the first large-scale sys- tematic study of candidate pretraining tasks, comparing 19 different tasks both as alter- natives and complements to language model- ing. Our primary results support the use lan- guage modeling, especially when combined with pretraining on additional labeled-data tasks. However, our results are mixed across pretraining tasks and show some concern- ing trends: In ELMo’s pretrain-then-freeze paradigm, random baselines are worryingly strong and results vary strikingly across tar- get tasks. In addition, fine-tuning BERT on an intermediate task often negatively impacts downstream transfer. In a more positive trend, we see modest gains from multitask training, suggesting the development of more sophis- ticated multitask and transfer learning tech- niques as an avenue for further research. 1 Introduction State-of-the-art models in natural language pro- cessing (NLP) often incorporate encoder func- tions which generate a sequence of vectors in- tended to represent the in-context meaning of each word in an input text. These encoders have typ- ically been trained directly on the target task at hand, which can be effective for data-rich tasks and yields human performance on some narrowly- defined benchmarks (Rajpurkar et al., 2018; Has- san et al., 2018), but is tenable only for the few tasks with millions of training data examples. This ∗This paper supercedes “Looking for ELMo’s Friends: Sentence-Level Pretraining Beyond Language Modeling”, an earlier version of this work by the same authors. Correspon- dence to: alexwang@nyu.edu Input Text Intermediate Task Model Intermediate Task Output BERT Input Text Intermediate Task Model Intermediate Task Output ELMo BiLSTM Input Text Target Task Model Target Task Output Intermediate Task-Trained BERT Input Text Pretraining Task Model Pretraining Task Output BiLSTM Input Text Target Task Model Target Task Output Pretrained BiLSTM Input Text Target Task Model Target Task Output ELMo Intermediate Task-Trained BiLSTM ❄ ❄ ❄ ❄ Figure 1: Learning settings that we consider. Model components with frozen parameters are shown in gray and decorated with snowflakes. Top (pretraining): We pretrain a BiLSTM on a task (left), and learn a target task model on top of the representations it produces (right). Middle (intermediate ELMo training): We train a BiLSTM on top of ELMo for an intermediate task (left). We then train a …

Content truncated to 500 words