Which *BERT? A Survey Organizing Contextualized Encoders

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7516–7533, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7516 Which *BERT? A Survey Organizing Contextualized Encoders Patrick Xia Shijie Wu Benjamin Van Durme Johns Hopkins University {paxia, vandurme}@cs.jhu.edu, shijie.wu@jhu.edu Abstract Pretrained contextualized text encoders are now a staple of the NLP community. We present a survey on language representation learning with the aim of consolidating a series of shared lessons learned across a variety of recent efforts. While significant advancements continue at a rapid pace, we find that enough has now been discovered, in different direc- tions, that we can begin to organize advances according to common themes. Through this organization, we highlight important consider- ations when interpreting recent contributions and choosing which model to use. 1 Introduction A couple years ago, Peters et al. (2018, ELMo) won the NAACL Best Paper Award for creating strong performing, task-agnostic sentence repre- sentations due to large scale unsupervised pretrain- ing. Days later, its high level of performance was surpassed by Radford et al. (2018) which boasted representations beyond a single sentence and fine- tuning flexibility. This instability and competition between models has been a recurring theme for researchers and practitioners who have watched the rapidly narrowing gap between text represen- tations and language understanding benchmarks. However, it has not discouraged research. Given the recent flurry of models, we often ask: “What, besides state-of-the-art, does this newest paper contribute? Which encoder should we use?” The goals of this survey are to outline the areas of progress, relate contributions in text encoders to ideas from other fields, describe how each area is evaluated, and present considerations for practi- tioners and researchers when choosing an encoder. This survey does not intend to compare specific model metrics, as tables from other works pro- vide comprehensive insight. For example, Table 16 in Raffel et al. (2019) compares the scores on a large suite of tasks of different model architectures, training objectives, and hyperparameters, and Ta- ble 1 in Rogers et al. (2020) details early efforts in model compression and distillation. We also rec- ommend other closely related surveys on contextu- alized word representations (Smith, 2019; Rogers et al., 2020; Liu et al., 2020a), transfer learning in NLP (Ruder et al., 2019), and integrating en- coders into NLP applications (Wolf et al., 2019). Complementing these existing bodies of work, we look at the ideas and progress in the scientific dis- course for text representations from the perspective of discerning their differences. We organize this paper as follows. §2 provides brief background on encoding, training, and eval- uating text representations. §3 identifies and ana- lyzes two classes of pretraining objectives. In §4, we explore faster and smaller models and architec- tures in both training and inference. §5 notes the impact of both quality and quantity of pretraining data. §6 briefly discusses efforts on probing en- coders and representations with respect to linguis- tic knowledge. §7 describes the efforts into training and evaluating multilingual representations. Within …

Content truncated to 500 words