Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4713–4722 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4713 Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy Colin B. Clement Microsoft Cloud and AI colin.clement@microsoft.com Shuai Lu Microsoft Research shuailu@microsoft.com Xiaoyu Liu Microsoft Cloud and AI lixiaoyu@microsoft.com Michele Tufano Microsoft Cloud and AI mitufano@microsoft.com Dawn Drain Microsoft Cloud and AI dawndrain95@gmail.com Nan Duan Microsoft Research nanduan@microsoft.com Neel Sundaresan Microsoft Cloud and AI neels@microsoft.com Alexey Svyatkovskiy Microsoft Cloud and AI alsvyatk@microsoft.com Abstract Statistical language modeling and translation with transformers have found many success- ful applications in program understanding and generation tasks, setting high benchmarks for tools in modern software development environ- ments. The finite context window of these neu- ral models means, however, that they will be unable to leverage the entire relevant context of large files and packages for any given task. While there are many efforts to extend the context window, we introduce an architecture- independent approach for leveraging the syn- tactic hierarchies of source code for incor- porating entire file-level context into a fixed- length window. Using concrete syntax trees of each source file we extract syntactic hier- archies and integrate them into context win- dow by selectively removing from view more specific, less relevant scopes for a given task. We evaluate this approach on code generation tasks and joint translation of natural language and source code in Python programming lan- guage, achieving a new state-of-the-art in code completion and summarization for Python in the CodeXGLUE benchmark. We also intro- duce new CodeXGLUE benchmarks for user- experience-motivated tasks: code completion with normalized literals, method body comple- tion/code summarization conditioned on file- level context. 1 Introduction Large transformer models (Vaswani et al., 2017) and the pre-training/fine-tuning paradigm (Devlin et al., 2018; Lewis et al., 2019; Radford et al., 2018) have become an essential part of state of the art natural language processing. Beyond the domain of natural language, these models and pro- cedures have enabled rapid progress in the soft- ware engineering space, including applications in code completion (Svyatkovskiy et al., 2020, 2019; Clement et al., 2020; Raychev et al., 2014; Bruch et al., 2009), natural language to code (NL2Code), code feature summarization (Clement et al., 2020; Moreno et al., 2013; Scalabrino et al., 2017; Wan et al., 2018; Alon et al., 2018; Moreno et al., 2014), code search (Husain et al., 2019; Feng et al., 2020), unit test generation (Tufano et al., 2020) and even bug fixing (Drain et al., 2021) and detection (Zhai et al., 2020). A major difference between transformers and their antecedents like recurrent neural networks (RNN) is their strictly enforced finite context win- dow. Whereas an RNN can iteratively consume as many tokens as is required, transformers can only consume up to a finite amount decided at training time. Further, it is impractical to simply expand the window as the memory and compute require- ments of the attention mechanism scale quadrati- cally with context …

Content truncated to 500 words