DIP: Dead code Insertion based Black-box Attack for Programming Language Model

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7777–7791 July 9-14, 2023 ©2023 Association for Computational Linguistics DIP: Dead code Insertion based Black-box Attack for Programming Language Model CheolWon Na, YunSeok Choi, Jee-Hyong Lee† College of Computing and Informatics Sungkyunkwan University Suwon, South Korea {ncw0034, ys.choi, john}@skku.edu Abstract Automatic processing of source code, such as code clone detection and software vulnerabil- ity detection, is very helpful to software en- gineers. Large pre-trained Programming Lan- guage (PL) models (such as CodeBERT, Graph- CodeBERT, CodeT5, etc.), show very powerful performance on these tasks. However, these PL models are vulnerable to adversarial exam- ples that are generated with slight perturbation. Unlike natural language, an adversarial exam- ple of code must be semantic-preserving and compilable. Due to the requirements, it is hard to directly apply the existing attack methods for natural language models. In this paper, we propose DIP (Dead code Insertion based Black- box Attack for Programming Language Model), a high-performance and efficient black-box at- tack method to generate adversarial examples using dead code insertion. We evaluate our proposed method on 9 victim downstream-task large code models. Our method outperforms the state-of-the-art black-box attack in both at- tack efficiency and attack quality, while gen- erated adversarial examples are compiled pre- serving semantic functionality. 1 Introduction Automatic processing of source code (such as clone detection, code completion, defect detection, etc.) is an important task that increases the productiv- ity of software engineers (Laguë et al., 1997; Li et al., 2006; Mockus, 2007; Kapser and Godfrey, 2008; Islam et al., 2016; Choi et al., 2021, 2023). For these tasks, deep learning-based models, such as code2seq (Alon et al., 2019a) and code2vec (Alon et al., 2019b), were developed. Recently, transformer-based large pre-trained programming language (PL) models (Feng et al., 2020; Guo et al., 2021b; Wang et al., 2021; Ahmad et al., 2021; Ding et al., 2021; Guo et al., 2022), showed powerful †Corresponding author. Figure 1: Dead code which is non-affecting to function- ality is inserted into the original code in order to mislead the target model. performance. However, these PL models are also vulnerable to adversarial examples as other natural language (NL) models (Choi et al., 2022; Zhang et al., 2020; Yang et al., 2022; Jha and Reddy, 2022; Jin et al., 2020; Li et al., 2020, 2021). Source code is textual data just like natural lan- guage, but the requirements of attack methods are different from NL model attacks. Attacks against NL models should consider semantic and fluency of original sentences, but attacks against PL models should preserve functionality and guarantee compi- lability. To consider for these issues, recent studies (Yefet et al., 2020; Yang et al., 2022; Srikant et al., 2021; Rabin et al., 2021; Zhang et al., 2020; Ra- makrishnan et al., 2020) use semantic-preserving code transformation which is commonly used in software engineering. The semantic-preserving code transformation includes Variable Renaming, Dead Code Insertion, Statement Permutation, Loop Exchange, Switch-to-If, and Boolean Exchange. Most …

Content truncated to 500 words