Life after BERT: What do Other Muppets Understand about Language?

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3180 - 3193 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Life after BERT: What do Other Muppets Understand about Language? Vladislav Lialin∗ Kevin Zhao∗ Namrata Shivagunde Anna Rumshisky Department of Computer Science University of Massachusetts Lowell vlialin@cs.uml.edu, kevin_zhao@uml.edu, shivagu@cs.uml.edu, arum@cs.uml.edu Abstract Existing pre-trained transformer analysis works usually focus only on one or two model families at a time, overlooking the variability of the architecture and pre-training objectives. In our work, we utilize the oLMpics bench- mark and psycholinguistic probing datasets for a diverse set of 29 models including T5, BART, and ALBERT. Additionally, we adapt the oLMpics zero-shot setup for autoregres- sive models and evaluate GPT networks of different sizes. Our findings show that none of these models can resolve compositional questions in a zero-shot fashion, suggesting that this skill is not learnable using existing pre-training objectives. Furthermore, we find that global model decisions such as architecture, directionality, size of the dataset, and pre-training objective are not predictive of a model’s linguistic capabilities. The code for this study is available on GitHub 1. 1 Introduction After the initial success of transfer learning in natu- ral language processing (Howard and Ruder, 2018; Peters et al., 2018), the number of pre-trained mod- els in NLP has increased dramatically (Radford and Narasimhan, 2018; Devlin et al., 2018; Lewis et al., 2019; Liu et al., 2019b; Raffel et al., 2019; Lan et al., 2019; Dong et al., 2019). However, there is a limited understanding of why certain models perform better than others and what linguistic ca- pabilities they acquire through pre-training. While a lot of work has been done to evaluate these models on general natural language under- standing datasets (Wang et al., 2018, 2019; Lai et al., 2017), such datasets do not allow researchers to identify the specific linguistic capabilities of a model. Furthermore the performance on these ∗The first two authors made equal contribu- tion to this work. Please direct correspondence to vlialin@cs.uml.edu, and kevin_zhao@uml.edu. 1github.com/kev-zhao/life-after-bert datasets results from a combination of pre-trained knowledge and task-specific information learned through fine-tuning. Probing tasks (Talmor et al., 2019; Zagoury et al., 2021; McCoy et al., 2019; Goldberg, 2019) give a promising solution to this problem, as they evalu- ate specific capabilities of pre-trained models, and in many cases, these tasks are designed for zero- shot evaluation, which reveals the knowledge that models have actually learned purely through the up- stream task. Currently, most in-depth analysis stud- ies focus on one or two model families. Many anal- ysis papers only probe BERT and similar models (Ettinger, 2020; Kobayashi et al., 2020; Garí Soler and Apidianaki, 2020; Ravichander et al., 2020; Zagoury et al., 2021; Kassner et al., 2020; Mohebbi et al., 2021; Clark et al., 2020; Liu et al., 2021). Fortunately, this trend is changing and now we see more papers that probe models such as ALBERT, T5 or BART (Mosbach et al., 2020; Phang et al., …

Content truncated to 500 words