Bootstrapped Unsupervised Sentence Representation Learning

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5168–5180 August 1–6, 2021. ©2021 Association for Computational Linguistics 5168 Bootstrapped Unsupervised Sentence Representation Learning Yan Zhang∗1 Ruidan He∗2 Zuozhu Liu†3 Lidong Bing2 Haizhou Li1,4 1National University of Singapore, Singapore 2DAMO Academy, Alibaba Group 3ZJU-UIUC Institute 4Kriston AI Lab, China eleyanz@nus.edu.sg, ruidan.he@alibaba-inc.com zuozhuliu@intl.zju.edu.cn, l.bing@alibaba-inc.com haizhou.li@nus.edu.sg Abstract As high-quality labeled data is scarce, unsu- pervised sentence representation learning has attracted much attention. In this paper, we propose a new framework with a two-branch Siamese Network which maximizes the simi- larity between two augmented views of each sentence. Specifically, given one augmented view of the input sentence, the online net- work branch is trained by predicting the rep- resentation yielded by the target network of the same sentence under another augmented view. Meanwhile, the target network branch is bootstrapped with a moving average of the online network. The proposed method signif- icantly outperforms other state-of-the-art un- supervised methods on semantic textual sim- ilarity (STS) and classification tasks. It can be adopted as a post-training procedure to boost the performance of the supervised meth- ods. We further extend our method for learn- ing multilingual sentence representations and demonstrate its effectiveness on cross-lingual STS tasks. Our code is available at https: //github.com/yanzhangnlp/BSL. 1 Introduction Sentence representation learning aims to map sen- tences into vectors that capture rich semantic in- formation. Among previous approaches, super- vised methods achieve state-of-the-art performance by leveraging quality sentence labels. For exam- ple, the recently proposed model Sentence-BERT (SBERT) (Reimers and Gurevych, 2019) fine-tunes a Siamese BERT network on natural language in- ference (NLI) tasks with labeled sentence pairs. It achieves state-of-the-art results on multiple seman- tic textual similarity (STS) tasks. However, such performance is mostly induced by high-quality su- pervision, while labeled data are difficult and ex- ∗ Equally Contributed. † Corresponding author. pensive to obtain in practice. Zhang et al. (2020) showed that SBERT generalizes poorly on target tasks that differ significantly from NLI on which SBERT is fine-tuned. Many unsupervised methods learn sentence representations by optimizing over various self- supervised learning (SSL) objectives on a large- scale unlabeled corpus. Early works often use auto-encoders (Socher et al., 2011; Hill et al., 2016) or next-sentence prediction (Kiros et al., 2015) for sentence representation learning. Re- cently, more efforts have been devoted to represen- tation learning with transformer-based networks us- ing masked language modeling (MLM). However, transformer-based methods do not directly produce meaningful sentence representations. Instead, sig- nificant supervised fine-tuning steps with labeled data are commonly required to form good represen- tations (Reimers and Gurevych, 2019). Recently, Giorgi et al. (2020) and Zhang et al. (2020) pro- posed novel transformer-based frameworks to di- rectly learn sentence representations from an unla- beled corpus, which even exhibited competitive per- formance to the supervised counterparts on some tasks. However, Giorgi et al. (2020) required long text during training while the contrastive learning strategy employed by Zhang et al. (2020) …

Content truncated to 500 words