Label Anchored Contrastive Learning for Language Understanding

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1437 - 1449 July 10-15, 2022 ©2022 Association for Computational Linguistics Label Anchored Contrastive Learning for Language Understanding Zhenyu Zhang, Yuming Zhao, Meng Chen, Xiaodong He JD AI, Beijing, China {zhangzhenyu47,zhaoyuming3,chenmeng20,xiaodong.he}@jd.com Abstract Contrastive learning (CL) has achieved aston- ishing progress in computer vision, speech, and natural language processing fields recently with self-supervised learning. However, CL approach to the supervised setting is not fully explored, especially for the natural language understanding classification task. Intuitively, the class label itself has the intrinsic abil- ity to perform hard positive/negative mining, which is crucial for CL. Motivated by this, we propose a novel label anchored contrastive learning approach (denoted as LaCon) for lan- guage understanding. Specifically, three con- trastive objectives are devised, including a multi-head instance-centered contrastive loss (ICL), a label-centered contrastive loss (LCL), and a label embedding regularizer (LER). Our approach does not require any specialized net- work architecture or any extra data augmenta- tion, thus it can be easily plugged into existing powerful pre-trained language models. Com- pared to the state-of-the-art baselines, LaCon obtains up to 4.1% improvement on the popu- lar datasets of GLUE and CLUE benchmarks. Besides, LaCon also demonstrates significant advantages under the few-shot and data imbal- ance settings, which obtains up to 9.4% im- provement on the FewGLUE and FewCLUE benchmarking tasks. 1 Introduction In recent years, contrastive learning (CL) has been widely applied to self-supervised representation learning and led to major advances across computer vision (CV) (He et al., 2019; Chen et al., 2020b), speech (Saeed et al., 2021; Chen et al., 2021), and natural language processing (NLP) (Fang and Xie, 2020; Gao et al., 2021; Yan et al., 2021). The ba- sic idea behind these works is to pull together an anchor and a “positive” sample in the embedding space, and to push apart the anchor from many “negative” samples. Since no labels are available, a positive pair often consists of data augmentations of the sample (a.k.a “views”), and negative pairs are formed by the anchor and randomly chosen sam- ples from the mini-batch. In visual representations, an effective solution to generate data augmenta- tions is to take two random transformations of the same image (e.g., cropping, flipping, distortion and rotation) (Chen et al., 2020b; Grill et al., 2020; Chen et al., 2020c). For natural language, simi- lar approaches are adopted such as word deletion, reordering, substitution, and back-translation etc. (Fang and Xie, 2020; Wang et al., 2021) However, data augmentation in NLP is inherently difficult because of its discrete nature. Therefore, some pre- vious works (Gao et al., 2021; Yan et al., 2021) also use dropout technique (Srivastava et al., 2014) to obtain sentence augmentations. Unlike self-supervised setting, some researchers propose supervised contrastive learning (SCL) (Khosla et al., 2020; Gunel et al., 2021; Suresh and Ong, 2021) which can construct positive pairs by leveraging label information. Examples from the same class are pulled closer than …

Content truncated to 500 words