What does the Failure to Reason with “Respectively” in Zero/Few-Shot Settings Tell Us about Language Models?

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 821–854 July 9-14, 2023 ©2023 Association for Computational Linguistics Multi-CLS BERT: An Efficient Alternative to Traditional Ensembling Haw-Shiuan Chang∗† Ruei-Yao Sun∗† Amazon USA chawshiu@amazon.com rueiyas@amazon.com Kathryn Ricci∗ Andrew McCallum CICS, UMass, Amherst 140 Governors Dr., Amherst, MA, USA kathryn.d.ricci@gmail.com mccallum@cs.umass.edu Abstract Ensembling BERT models often significantly improves accuracy, but at the cost of signifi- cantly more computation and memory footprint. In this work, we propose Multi-CLS BERT, a novel ensembling method for CLS-based pre- diction tasks that is almost as efficient as a sin- gle BERT model. Multi-CLS BERT uses mul- tiple CLS tokens with a parameterization and objective that encourages their diversity. Thus instead of fine-tuning each BERT model in an ensemble (and running them all at test time), we need only fine-tune our single Multi-CLS BERT model (and run the one model at test time, ensembling just the multiple final CLS embeddings). To test its effectiveness, we build Multi-CLS BERT on top of a state-of-the-art pretraining method for BERT (Aroca-Ouellette and Rudzicz, 2020). In experiments on GLUE and SuperGLUE we show that our Multi-CLS BERT reliably improves both overall accuracy and confidence estimation. When only 100 training samples are available in GLUE, the Multi-CLS BERTBase model can even outper- form the corresponding BERTLarge model. We analyze the behavior of our Multi-CLS BERT, showing that it has many of the same charac- teristics and behavior as a typical BERT 5-way ensemble, but with nearly 4-times less compu- tation and memory. 1 Introduction BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al., 2019) is one of the most widely-used language model (LM) archi- tectures for natural language understanding (NLU) tasks. We often fine-tune the pretrained BERT or its variants such as RoBERTa (Liu et al., 2019) so that the LMs learn to aggregate all the contextualized word embeddings into a single CLS embedding for a downstream text classification task. ∗indicates equal contribution †The work is done while the authors were at UMass BERT VS ([CLS] + BERT)*5 [CLS]*5 + BERT Proposed Multi-CLS BERT Classic 5 BERT Ensemble LC S BERT LC S Fine-tuning: Once Inference: Once Fine-tuning: 5 Times Inference: 5 Times Figure 1: Comparison of Multi-CLS BERT and the clas- sic BERT ensemble. Multi-CLS BERT only ensembles the multiple CLS embeddings in one BERT encoder rather than ensemble multiple BERT encoders with dif- ferent parameter weights. During fine-tuning, different initializations and different training data orders significantly affect BERT’s generalization performance, especially with a small training dataset (Dodge et al., 2020; Zhang et al., 2021a; Mosbach et al., 2021). One simple and popular solution to the issue is to fine- tune BERT model multiple times using different random seeds and ensemble their predictions to improve its accuracy and confidence estimation. Although very effective, the memory and compu- tational cost of ensembling a large LM is often prohibitive (Xu et al., 2020; Liang et al., 2022). Naturally, we would like to ask, “Is it …

Content truncated to 500 words