ReACC: A Retrieval-Augmented Code Completion Framework

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6227 - 6240 May 22-27, 2022 c⃝2022 Association for Computational Linguistics ReACC: A Retrieval-Augmented Code Completion Framework Shuai Lu1, Nan Duan1, Hojae Han2∗, Daya Guo3∗, Seung-won Hwang2, Alexey Svyatkovskiy4 1Microsoft Research Asia 3Sun Yat-sen University2Seoul National University 4Microsoft Devdiv {shuailu,nanduan,alsvyatk}@microsoft.com {stovecat,seungwonh}@snu.ac.kr guody5@mail2.sysu.edu.cn Abstract Code completion, which aims to predict the following code token(s) according to the code context, can improve the productivity of soft- ware development. Recent work has proved that statistical language modeling with trans- formers can greatly improve the performance in the code completion task via learning from large-scale source code datasets. However, current approaches focus only on code context within the file or project, i.e. internal context. Our distinction is utilizing “external” context, inspired by human behaviors of copying from the related code snippets when writing code. Specifically, we propose a retrieval-augmented code completion framework, leveraging both lexical copying and referring to code with sim- ilar semantics by retrieval. We adopt a stage- wise training approach that combines a source code retriever and an auto-regressive language model for programming language. We evalu- ate our approach in the code completion task in Python and Java programming languages, achieving a state-of-the-art performance on CodeXGLUE benchmark. 1 Introduction With the growth of software engineering field, large-scale source code corpus gives a chance to train language models in code domain (Hindle et al., 2016; Tu et al., 2014). And benefiting from the large transformer models (Vaswani et al., 2017) and pre-training techniques (Devlin et al., 2018; Radford et al., 2018), a rapid progress has been made in many code-related tasks like code search (Feng et al., 2020; Guo et al., 2020), code summa- rization (Clement et al., 2020; Ahmad et al., 2020), bug fixing (Mashhadi and Hemmati, 2021; Drain et al., 2021) and code completion (Svyatkovskiy et al., 2020; Liu et al., 2020; Kim et al., 2021; Clement et al., 2020). ∗Work done during internship at Microsoft. Code completion is considered as an essential feature towards efficient software development in modern Integrated Development Environments (IDEs). The task is formulated by predicting the following code token(s) based on the code context. Traditionally, code completion requires real-time program analysis and recommends type-correct code tokens (Tu et al., 2014). Recently, statisti- cal language models trained on large-scale source code data have shown high accuracy in the code completion task. Primitive approaches take the given context only (Liu et al., 2016; Karampatsis et al., 2020), some methods use richer information, e.g., adding code token types (Liu et al., 2020), ab- stract syntax tree (AST) structures (Li et al., 2018; Kim et al., 2021), or extended hierarchical context (Clement et al., 2021). However, one key limita- tion of existing methods is the scope of information they utilize; all the information is bounded in the given input file. This is unnatural from human per- spective, as studies demonstrate that programmers tend to reuse an existing code snippet by copying part …

Content truncated to 500 words