CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8696–8708 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8696 CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation Yue Wang1, Weishi Wang12, Shafiq Joty12, and Steven C.H. Hoi1 1 Salesforce Research Asia 2 Nanyang Technological University, Singapore {wang.y,weishi.wang,sjoty,shoi}@salesforce.com Abstract Pre-trained models for Natural Languages (NL) like BERT and GPT have been recently shown to transfer well to Programming Lan- guages (PL) and largely benefit a broad set of code-related tasks. Despite their success, most current methods either rely on an encoder-only (or decoder-only) pre-training that is subopti- mal for generation (resp. understanding) tasks or process the code snippet in the same way as NL, neglecting the special characteristics of PL such as token types. We present CodeT5, a unified pre-trained encoder-decoder Trans- former model that better leverages the code se- mantics conveyed from the developer-assigned identifiers. Our model employs a unified framework to seamlessly support both code un- derstanding and generation tasks and allows for multi-task learning. Besides, we propose a novel identifier-aware pre-training task that en- ables the model to distinguish which code to- kens are identifiers and to recover them when they are masked. Furthermore, we propose to exploit the user-written code comments with a bimodal dual generation task for better NL-PL alignment. Comprehensive experiments show that CodeT5 significantly outperforms prior methods on understanding tasks such as code defect detection and clone detection, and gen- eration tasks across various directions includ- ing PL-NL, NL-PL, and PL-PL. Further analy- sis reveals that our model can better capture semantic information from code. Our code and pre-trained models are released at https: //github.com/salesforce/CodeT5. 1 Introduction Pre-trained language models such as BERT (De- vlin et al., 2019), GPT (Radford et al., 2019), and T5 (Raffel et al., 2020) have greatly boosted per- formance in a wide spectrum of natural language processing (NLP) tasks. They typically employ a pre-train then fine-tune paradigm that aims to derive generic language representations by self- supervised training on large-scale unlabeled data, which can be transferred to benefit multiple down- stream tasks, especially those with limited data an- notation. Inspired by their success, there are many recent attempts to adapt these pre-training meth- ods for programming language (PL) (Svyatkovskiy et al., 2020; Kanade et al., 2020; Feng et al., 2020), showing promising results on code-related tasks. However, despite their success, most of these models rely on either an encoder-only model simi- lar to BERT (Svyatkovskiy et al., 2020; Feng et al., 2020) or a decoder-only model like GPT (Kanade et al., 2020), which is suboptimal for generation and understanding tasks, respectively. For exam- ple, CodeBERT (Feng et al., 2020) requires an additional decoder when applied for the code sum- marization task, where this decoder cannot bene- fit from the pre-training. Besides, most existing methods simply employ the conventional NLP pre- training techniques on source code by regarding it as a sequence of tokens like NL. This largely ig- nores the rich structural …

Content truncated to 500 words