Don’t Judge a Language Model by Its Last Layer: Contrastive Learning with Layer-Wise Attention Pooling

Proceedings of the 29th International Conference on Computational Linguistics, pages 4585–4592 October 12–17, 2022. 4585 Don’t Judge a Language Model by Its Last Layer: Contrastive Learning with Layer-Wise Attention Pooling Dongsuk Oh∗‡, Yejin Kim∗§, Hodong Lee‡, H. Howie Huang† § and Heuiseok Lim† ‡ ‡Computer Science and Engineering, Korea University §Graph Lab., George Washington University {inow3555,bigshane319,limhseok}@korea.ac.kr {yejinjenny, howie}@gwu.edu Abstract Recent pre-trained language models (PLMs) achieved great success on many natural lan- guage processing tasks through learning lin- guistic features and contextualized sentence representation. Since attributes captured in stacked layers of PLMs are not clearly iden- tified, straightforward approaches such as em- bedding the last layer are commonly preferred to derive sentence representations from PLMs. This paper introduces the attention-based pool- ing strategy, which enables the model to pre- serve layer-wise signals captured in each layer and learn digested linguistic features for down- stream tasks. The contrastive learning objec- tive can adapt the layer-wise attention pooling to both unsupervised and supervised manners. It results in regularizing the anisotropic space of pre-trained embeddings and being more uni- form. We evaluate our model on standard se- mantic textual similarity (STS) and seman- tic search tasks. As a result, our method im- proved the performance of the base contrastive learned BERTbase and variants. 1 Introduction Pre-trained language models (PLMs) (Kenton and Toutanova, 2019; Liu et al., 2019; Radford et al., 2019; Raffel et al., 2019) have shown competi- tive performance on many natural language pro- cessing (NLP) tasks. Also, contrastive learning us- ing the PLMs shows the highest performance in sentence representation. Contrastive learning is to learn effective representations by staying semanti- cally close sample pairs together while dissimilar ones are far apart(Hadsell et al., 2006). In general, PLMs use either [CLS] tokens in the last layer, AV G which is the average representa- tion of tokens in the last layer(Reimers et al., 2019; Li et al., 2020), or AV GFL which is the average * These authors contributed equally. † These authors are corresponding authors. Figure 1: Spearsman’s correlation score of each layer evaluated on STS-B test set Figure 2: Spearsman’s correlation score depending on the pooling methods of PLMs for each domain representation of tokens in the first and last lay- ers(Gao et al., 2021), to pool out sentence represen- tation from word representations. However, since language models show performance gaps by do- main when trained on different objectives, the fixed pooling strategy has limitations in performance im- provement. Figure 1 and 2 show the Spearman’s correlation score of each layer or pooling method in PLMs. We evaluated the test set of the standard semantic textual similarity (STS) dataset(Cer et al., 2017; Agirre et al., 2012, 2013, 2014, 2015, 2016; Marelli 4586 et al., 2014). The comparison of performance when pooling each layer shown in Figure 1 indicates that using only a specific layer for pooling is insufficient. Other layers other than the last layer may con- tain substantial information for sentence represen- tation. For example, for the STS benchmark (STS- B) …

Content truncated to 500 words