An End-to-End Contrastive Self-Supervised Learning Framework for Language Understanding

An End-to-End Contrastive Self-Supervised Learning Framework for Language Understanding Hongchao Fang, Pengtao Xie∗ University of California San Diego, USA p1xie@eng.ucsd.edu Abstract Self-supervised learning (SSL) methods such as Word2vec, BERT, and GPT have shown great effectiveness in language understanding. Contrastive learning, as a recent SSL approach, has attracted increasing attention in NLP. Con- trastive learning learns data representations by predicting whether two augmented data in- stances are generated from the same original data example. Previous contrastive learning methods perform data augmentation and con- trastive learning separately. As a result, the augmented data may not be optimal for con- trastive learning. To address this problem, we propose a four-level optimization framework that performs data augmentation and contras- tive learning end-to-end, to enable the aug- mented data to be tailored to the contrastive learning task. This framework consists of four learning stages, including training machine translation models for sentence augmentation, pretraining a text encoder using contrastive learning, finetuning a text classification model, and updating weights of translation data by minimizing the validation loss of the classifi- cation model, which are performed in a unified way. Experiments on datasets in the GLUE benchmark (Wang et al., 2018a) and on da- tasets used in Gururangan et al. (2020) dem- onstrate the effectiveness of our method. 1 Introduction Self-supervised learning (Bengio et al., 2000; Mikolov et al., 2013; Devlin et al., 2019; Radford et al., 2018; Lewis et al., 2020), which learns data representations by solving prediction tasks defined on input data without leveraging human- provided labels, has achieved broad success in NLP. Many NLP-specific self-supervised learn- ing (SSL) methods have been proposed, such as neural language models (Bengio et al., 2000), Word2vec (Mikolov et al., 2013), BERT (Devlin et al., 2019), GPT (Radford et al., 2018), BART ∗Corresponding author. (Lewis et al., 2020), and so forth, with various SSL tasks defined. For example, in Word2vec and BERT, the SSL task is predicting the identities of masked tokens based on their contexts. In neu- ral language models including GPT, the SSL task is language modeling: Given a history of tokens, predict the next token. Recently, contrastive self-supervised learning (He et al., 2020; Chen et al., 2020) has been borrowed from vision domains into NLP and has shown promising success in predicting seman- tic textual similarity (Gao et al., 2021), machine translation (Pan et al., 2021), relation extraction (Su et al., 2021), and so on. The key idea of con- trastive self-supervised learning (CSSL) is: Create augments of original examples, then learn repre- sentations by predicting whether two augments are from the same original data example. In ex- isting CSSL approaches, data augmentation and contrastive learning are performed separately. As a result, augmented data may not be optimal for contrastive learning. For example, considering a back-translation (Sennrich et al., 2016)–based augmentation method, if the translation model is trained using news corpora, it is not suitable for augmenting data for movie review data. In this paper, we aim to address this issue. We propose a four-level optimization framework …

Content truncated to 500 words