ContraCLM: Contrastive Learning For Causal Language Model

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6436–6459 July 9-14, 2023 ©2023 Association for Computational Linguistics CONTRACLM: Contrastive Learning For Causal Language Model Nihal Jain∗, Dejiao Zhang∗, Wasi Uddin Ahmad∗, Zijian Wang, Feng Nan, Xiaopeng Li, Ming Tan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Xiaofei Ma, Bing Xiang AWS AI Labs, USA Abstract Despite exciting progress in causal language models, the expressiveness of their representa- tions is largely limited due to poor discrimina- tion ability. To remedy this issue, we present CONTRACLM, a novel contrastive learning framework at both the token-level and the sequence-level. We assess CONTRACLM on a variety of downstream tasks. We show that CONTRACLM enhances the discrimination of representations and bridges the gap with encoder-only models, which makes causal lan- guage models better suited for tasks beyond language generation. Specifically, we attain 44% relative improvement on the Semantic Tex- tual Similarity tasks and 34% on Code-to-Code Search tasks. Furthermore, by improving the expressiveness of representations, CONTRA- CLM also boosts the source code generation capability with 9% relative improvement on execution accuracy on the HumanEval bench- mark. 1 1 Introduction Causal Language Models (CLM) have seen remark- able success in language generation, both in natural language (Radford et al., 2018, 2019; Brown et al., 2020) and programming language (Chen et al., 2021; Nijkamp et al., 2022). However, one limita- tion at their core is the poor discrimination ability of the representations, which often causes a large performance gap with encoder-only or encoder- decoder models on discriminative tasks (see Ap- pendix D.1), and hence limits the wide usage of CLM beyond language generation. Prior studies posit that the anisotropy issue, i.e., representations being squeezed into a tiny cone in the vector space (Ethayarajh, 2019), can be the main cause of the poor discrimination ability of ∗Equal Contribution. Correspondence to Dejiao Zhang <dejiaoz@amazon.com>. 1We release our code at https://github.com/ amazon-science/ContraCLM. language models across different architectures and objectives. Many efforts have focused on resolving the anisotropy issue on encoder-only or encoder- decoder models, either through post-processing (Su et al., 2021; Li et al., 2020) or integrating differ- ent regularization terms into the training objective (Gao et al., 2019; Wang et al., 2020). A recent work (Su and Collier, 2022) shows that the decoder-only CLM does not suffer from the anisotropic prob- lem as long as the model is beyond a certain size. However, we find that such conclusions can vary across domains. As shown in Figure 1a, CLMs pre- trained on text, i.e., GPT-2 (Radford et al., 2019), do yield representations with good isotropy and dis- crimination as long as the model is not smaller than 774M parameters (GPT2-Large), whilst CodeGen (Nijkamp et al., 2022), pretrained on programming language data, consistently suffers from anisotropy and poor discrimination across different model sizes. Therefore, an effective training strategy is still essential for CLMs to improve representation quality with better isotropy and discrimination (Fig- ure 1b). We conjecture that this is essential not only …

Content truncated to 500 words