Towards Learning (Dis)-Similarity of Source Code from Program Contrasts

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6300 - 6312 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Towards Learning (Dis)-Similarity of Source Code from Program Contrasts Yangruibo Ding§, Luca Buratti†, Saurabh Pujar†, Alessandro Morari†, Baishakhi Ray§, and Saikat Chakraborty§ § Columbia University † IBM Research §{yrbding, rayb, saikatc}@cs.columbia.edu †{luca.buratti1, saurabh.pujar}@ibm.com, amorari@us.ibm.com Abstract Understanding the functional (dis)-similarity of source code is significant for code model- ing tasks such as software vulnerability and code clone detection. We present DISCO (DIS- similarity of COde), a novel self-supervised model focusing on identifying (dis)similar func- tionalities of source code. Different from ex- isting works, our approach does not require a huge amount of randomly collected datasets. Rather, we design structure-guided code trans- formation algorithms to generate synthetic code clones and inject real-world security bugs, aug- menting the collected datasets in a targeted way. We propose to pre-train the Transformer model with such automatically generated program con- trasts to better identify similar code in the wild and differentiate vulnerable programs from be- nign ones. To better capture the structural fea- tures of source code, we propose a new cloze objective to encode the local tree-based context (e.g., parents or sibling nodes). We pre-train our model with a much smaller dataset, the size of which is only 5% of the state-of-the-art models’ training datasets, to illustrate the ef- fectiveness of our data augmentation and the pre-training approach. The evaluation shows that, even with much less data, DISCO can still outperform the state-of-the-art models in vulnerability and code clone detection tasks. 1 Introduction Understanding the functional similar- ity/dissimilarity of source code is at the core of several code modeling tasks such as software vulnerability and code clone detection, which are important for software maintenance (Kim et al., 2017; Li et al., 2016). Existing pre-trained Transformer models (Guo et al., 2021; Feng et al., 2020; Ahmad et al., 2021) show promises for understanding code syntax (i.e., tokens and structures). However, they still get confused when trying to identify functional (dis)-similarities. For instance, syntax-based models can embed two code fragments with identical functionality but very different tokens and structures as distinct vectors and fail to identify them as semantically similar. Likewise, these models cannot distinguish between two code fragments that differ in functionalities but share a close syntactic resemblance. For example, consider an if statement if(len(buf) < N) checking buffer length before accessing the buffer. Keeping the rest of the program the same, if we simply replace the token ‘<’ with ‘≤,’ the modifi- cation can potentially trigger security vulnerability, e.g., buffer overflow bug1. It is challenging for existing pre-training techniques to tell apart such subtle differences in the functionalities. In addition, existing pre-training techniques rely on a huge volume of training corpus that is ran- domly selected. For fine-tuning tasks like code clone detection or vulnerability detection, such ran- dom selection of training data is never tailored to teach the model about code functionalities. To address these limitations, …

Content truncated to 500 words