Bi-Granularity Contrastive Learning for Post-Training in Few-Shot Scene

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1733–1742 August 1–6, 2021. ©2021 Association for Computational Linguistics 1733 Bi-Granularity Contrastive Learning for Post-Training in Few-Shot Scene Ruikun Luo, Guanhuan Huang, Xiaojun Quan∗ School of Computer Science and Engineering, Sun Yat-sen University, China {luork, huanggh25}@mail2.sysu.edu.cn, quanxj3@mail.sysu.edu.cn Abstract The major paradigm of applying a pre-trained language model to downstream tasks is to fine- tune it on labeled task data, which often suf- fers instability and low performance when the labeled examples are scarce. One way to alle- viate this problem is to apply post-training on unlabeled task data before fine-tuning, adapt- ing the pre-trained model to target domains by contrastive learning that considers either token- level or sequence-level similarity. Inspired by the success of sequence masking, we argue that both token-level and sequence-level similari- ties can be captured with a pair of masked se- quences. Therefore, we propose complemen- tary random masking (CRM) to generate a pair of masked sequences from an input se- quence for sequence-level contrastive learning and then develop contrastive masked language modeling (CMLM) for post-training to inte- grate both token-level and sequence-level con- trastive learnings. Empirical results show that CMLM surpasses several recent post-training methods in few-shot settings without the need for data augmentation. 1 Introduction The past few years have seen the rapid proliferation of large-scale pre-trained language models such as GPT (Radford et al., 2018), BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019b). These models are generally characterized by pre-training on huge general-domain corpora and then fine- tuning on task-specific labeled examples when ap- plied to a downstream task. Despite tremendous knowledge obtained from general-domain corpora through pre-training, sufficient labeled examples from the task domain are still needed. However, collecting them is often infeasible in many scenes, where it tends to be unstable and low-performing ∗Corresponding author. it 's a charming and often [M] journey affecting it 's [M] charming [M] [M] affecting journey Sequence-Level Similarity Token-Level Similarity Figure 1: Demonstration of token-level similarity and sequence-level similarity, where the token “affecting” is close to the token “[MASK]” due to the latter’s context. The two sequences with different maskings are also close due to their semantic affinity. when directly fine-tuning these pre-trained models (Zhang et al., 2021; Dodge et al., 2020). Many efforts have been devoted to addressing the above issue. Firstly, it can be relieved by im- proving the fine-tuning process, such as introduc- ing regularization (Jiang et al., 2020; Lee et al., 2020), re-initializing top layers (Zhang et al., 2021), and using debiased Adam optimizer (Mosbach et al., 2021). Besides, according to empirical re- sults (Zhang et al., 2021; Mosbach et al., 2021), fine-tuning with a small learning rate and more fine- tuning epochs can also improve the situation. Sec- ondly, additional data can be explored, for which two main genres of data might be helpful: labeled examples from related tasks and unlabeled task examples. The former has shown considerable suc- cess on the GLUE tasks (Phang et al., 2018; …

Content truncated to 500 words