ElitePLM: An Empirical Study on General Language Ability Evaluation of Pretrained Language Models

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3519 - 3539 July 10-15, 2022 ©2022 Association for Computational Linguistics ElitePLM: An Empirical Study on General Language Ability Evaluation of Pretrained Language Models Junyi Li1,5, Tianyi Tang1, Zheng Gong1, Lixin Yang2, Zhuohao Yu2, Zhipeng Chen2, Jingyuan Wang3,4, Wayne Xin Zhao1,5∗and Ji-Rong Wen1,5 1Gaoling School of Artificial Intelligence, Renmin University of China 2Renmin University of China 3Peng Cheng Laboratory 4School of Computer Science and Engineering, Beihang University 5Beijing Key Laboratory of Big Data Management and Analysis Methods {lijunyi,steven_tang}@ruc.edu.cn batmanfly@gmail.com Abstract Nowadays, pretrained language models (PLMs) have dominated the majority of NLP tasks. While, little research has been conducted on systematically evaluating the language abilities of PLMs. In this paper, we present a large-scale empirical study on genEral language ability evaluation of PLMs (ElitePLM). In our study, we design four evaluation dimensions, i.e., memory, comprehension, reasoning, and com- position, to measure ten widely-used PLMs within five categories. Our empirical results demonstrate that: (1) PLMs with varying train- ing objectives and strategies are good at differ- ent ability tests; (2) fine-tuning PLMs in down- stream tasks is usually sensitive to the data size and distribution; (3) PLMs have excellent trans- ferability between similar tasks. Moreover, the prediction results of PLMs in our experiments are released as an open resource for more deep and detailed analysis on the language abilities of PLMs. This paper can guide the future work to select, apply, and design PLMs for specific tasks. We have made all the details of experiments publicly available at https: //github.com/RUCAIBox/ElitePLM. 1 Introduction Recent years have featured a trend towards Trans- former (Vaswani et al., 2017) based pretrained lan- guage models (PLMs) in natural language process- ing (NLP) systems. By being pretrained on massive unlabeled text, PLMs can be directly fine-tuned on downstream tasks, entirely removing the need for task-specific architectures (Radford et al., 2018). This paradigm has led to significant progress on many challenging NLP tasks such as reading com- prehension (Devlin et al., 2019) and text genera- tion (Brown et al., 2020). With rising new state-of-the-art results that ap- proach or surpass human performance on several tasks, it is a non-trivial research topic about how ∗Corresponding author to systematically evaluate the language abilities of PLMs from a wide range of perspectives. Given a wide range of publicly released PLMs, it is partic- ularly useful to derive principles or guidelines for selecting suitable PLMs for specific downstream tasks. However, existing works either target some single ability (Talmor et al., 2020; Zhou et al., 2020), or consider a simple mixture of multiple (small-scale) tasks that lack a comprehensive de- sign and test (Wang et al., 2019b; Liang Xu, 2020). There has been no detailed and systematic analysis of PLM’s abilities in large-scale NLP tasks. To fill the gap of PLMs evaluation, we introduce the genEral language ability evaluation (ElitePLM) for empirically and systematically assessing the general language abilities of PLMs. The ideal goal …

Content truncated to 500 words