Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12933–12950 July 9-14, 2023 ©2023 Association for Computational Linguistics Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers Linyuan Gong1∗, Chenyan Xiong2, Xiaodong Liu2, Payal Bajaj2, Yiqing Xie3∗, Alvin Cheung1, Jianfeng Gao2 and Xia Song2 1UC Berkeley, 2Microsoft, 3Carnegie Mellon University 1{gly,akcheung}@berkeley.edu 2{chenyan.xiong,xiaodl,payal.bajaj,jfgao,xiaso}@microsoft.com 3yiqingxi@andrew.cmu.edu Abstract This paper explores the effectiveness of model- generated signals in improving zero-shot gen- eralization of text-to-text Transformers such as T5. We study various designs to pretrain T5 using an auxiliary model to construct more chal- lenging token replacements for the main model to denoise. Key aspects under study include the decoding target, the location of the RTD head, and the masking pattern. Based on these studies, we develop a new model, METRO- T0, which is pretrained using the redesigned ELECTRA-Style pretraining strategies and then prompt-finetuned on a mixture of NLP tasks. METRO-T0 outperforms all similar- sized baselines on prompted NLP benchmarks, such as T0 Eval and MMLU, and rivals the state-of-the-art T011B model with only 8% of its parameters. Our analysis on model’s neural activation and parameter sensitivity reveals that the effectiveness of METRO-T0 stems from more balanced contribution of parameters and better utilization of their capacity. The code and model checkpoints are available at https: //github.com/gonglinyuan/metro_t0. 1 Introduction Recent work in NLP has shown that pretrained lan- guage models have made noteworthy progress to- ward generalization to unseen tasks. Despite being pretrained on only language modeling objectives, large language models can perform reasonable zero- shot generalization given natural language instruc- tions, i.e. prompts (Radford et al., 2019; Brown et al., 2020). Further research shows that finetuning language models on a mixture of tasks with prompt templates enhances their performance on held-out new tasks (Sanh et al., 2022; Wei et al., 2021). In recent years, two significant research paths have emerged in the field of pretrained language ∗Part of this work is done during Linyuan and Yiqing’s internship at Microsoft. models: one seeks to improve generalization ei- ther by scaling up the model, increasing param- eters, data, and compute, or by refining prompts. Another divergent yet complementary approach focuses on augmenting the efficiency of pretrain- ing, particularly in the context of BERT-style mod- els. This approach has been proven to signifi- cantly improve pretraining efficiency through the use of model-generated pretraining signals, as evi- denced by ELECTRA (Clark et al., 2020), COCO- LM (Meng et al., 2021), and METRO-LM (Bajaj et al., 2022). However, this improvement has pri- marily been witnessed in single-task supervised finetuning settings. Our work seeks to bridge these two areas of research. We present a novel method that enhances the pretraining efficiency of T5, a widely used encoder-decoder Transformer in prompt-based learning, by utilizing ELECTRA- Style model-generated signals. Our preliminary studies, however, encountered many challenges in pretraining T5 with model- generated signals, particularly in designing an ef- fective objective to train the decoder and ensuring training stability. To address these challenges, we study the …

Content truncated to 500 words