CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2332–2342 August 1–6, 2021. ©2021 Association for Computational Linguistics 2332 CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding Dong Wang1,2∗, Ning Ding1,2∗, Piji Li3† , Hai-Tao Zheng1,2† 1Department of Computer Science and Technology, Tsinghua University 2Tsinghua ShenZhen International Graduate School, Tsinghua University 3Tencent AI Lab {wangd18,dingn18}@mails.tsinghua.edu.cn pijili@tencent.com,zheng.haitao@sz.tsinghua.edu.cn Abstract Despite pre-trained language models have proven useful for learning high-quality seman- tic representations, these models are still vul- nerable to simple perturbations. Recent works aimed to improve the robustness of pre-trained models mainly focus on adversarial training from perturbed examples with similar seman- tics, neglecting the utilization of different or even opposite semantics. Different from the image processing field, the text is discrete and few word substitutions can cause significant se- mantic changes. To study the impact of seman- tics caused by small perturbations, we conduct a series of pilot experiments and surprisingly find that adversarial training is useless or even harmful for the model to detect these semantic changes. To address this problem, we propose Contrastive Learning with semantIc Negative Examples (CLINE), which constructs seman- tic negative examples unsupervised to improve the robustness under semantically adversarial attacking. By comparing with similar and op- posite semantic examples, the model can ef- fectively perceive the semantic changes caused by small perturbations. Empirical results show that our approach yields substantial improve- ments on a range of sentiment analysis, reason- ing, and reading comprehension tasks. And CLINE also ensures the compactness within the same semantics and separability across dif- ferent semantics in sentence-level. 1 Introduction Pre-trained language models (PLMs) such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have been proved to be an effective way to improve various natural language processing tasks. However, recent works show that PLMs suffer from ∗ Equal contribution. This work was mainly done when Dong Wang was an intern at Tencent AI Lab. † Corresponding authors. Sentence Label Predict creepy but ultimately unsatisfying thriller Negative Negative creepy but lastly unsat- isfying thriller Negative Positive creepy but ultimately satisfying thriller Positive Negative Table 1: An adversarial example of sentiment analysis in movie reviews. And the prediction results are from the BERT (base version with 12 layers). poor robustness when encountering adversarial ex- amples (Jin et al., 2020; Li et al., 2020; Garg and Ramakrishnan, 2020; Zang et al., 2020; Lin et al., 2020a). As shown in Table 1, the BERT model can be fooled easily just by replacing ultimately with a similar word lastly. To improve the robustness of PLMs, recent stud- ies attempt to adopt adversarial training on PLMs, which applies gradient-based perturbations to the word embeddings during training (Miyato et al., 2017; Zhu et al., 2020; Jiang et al., 2020) or adds high-quality adversarial textual examples to the training phase (Wang and Bansal, 2018; Michel et al., 2019). The primary goal of these adversarial methods is to keep …

Content truncated to 500 words