Self-Guided Contrastive Learning for BERT Sentence Representations

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2528–2540 August 1–6, 2021. ©2021 Association for Computational Linguistics 2528 Self-Guided Contrastive Learning for BERT Sentence Representations Taeuk Kim†∗, Kang Min Yoo‡, and Sang-goo Lee† †Dept. of Computer Science and Engineering, Seoul National University, Seoul, Korea ‡NAVER AI Lab, Seongnam, Korea {taeuk,sglee}@europa.snu.ac.kr, kangmin.yoo@navercorp.com Abstract Although BERT and its variants have reshaped the NLP landscape, it still remains unclear how best to derive sentence embeddings from such pre-trained Transformers. In this work, we propose a contrastive learning method that utilizes self-guidance for improving the qual- ity of BERT sentence representations. Our method fine-tunes BERT in a self-supervised fashion, does not rely on data augmentation, and enables the usual [CLS] token embed- dings to function as sentence vectors. More- over, we redesign the contrastive learning ob- jective (NT-Xent) and apply it to sentence rep- resentation learning. We demonstrate with ex- tensive experiments that our approach is more effective than competitive baselines on diverse sentence-related tasks. We also show it is effi- cient at inference and robust to domain shifts. 1 Introduction Pre-trained Transformer (Vaswani et al., 2017) lan- guage models such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have been integral to achieving recent improvements in natural lan- guage understanding. However, it is not straightfor- ward to directly utilize these models for sentence- level tasks, as they are basically pre-trained to focus on predicting (sub)word tokens given context. The most typical way of converting the models into sen- tence encoders is to fine-tune them with supervision from a downstream task. In the process, as initially proposed by Devlin et al. (2019), a pre-defined to- ken’s (a.k.a. [CLS]) embedding from the last layer of the encoder is deemed as the representation of an input sequence. This simple but effective method is possible because, during supervised fine-tuning, the [CLS] embedding functions as the only com- munication gate between the pre-trained encoder *This work has been mainly conducted when TK was a research intern at NAVER AI Lab. Figure 1: BERT(-base)’s layer-wise performance with different pooling methods on the STS-B test set. We observe that the performance can be dramatically var- ied according to the selected layer and pooling strategy. Our self-guided training (SG / SG-OPT) assures much improved results compared to those of the baselines. and a task-specific layer, encouraging the [CLS] vector to capture the holistic information. On the other hand, in cases where labeled datasets are unavailable, it is unclear what the best strategy is for deriving sentence embeddings from BERT.1 In practice, previous studies (Reimers and Gurevych, 2019; Li et al., 2020; Hu et al., 2020) reported that na¨ıvely (i.e., without any processing) leveraging the [CLS] embedding as a sentence representation, as is the case of supervised fine- tuning, results in disappointing outcomes. Cur- rently, the most common rule of thumb for building BERT sentence embeddings without supervision is to apply mean pooling …

Content truncated to 500 words