Unified Pre-training for Program Understanding and Generation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655–2668 June 6–11, 2021. ©2021 Association for Computational Linguistics 2655 Unified Pre-training for Program Understanding and Generation Wasi Uddin Ahmad§∗, Saikat Chakraborty†∗, Baishakhi Ray†, Kai-Wei Chang§ §University of California, Los Angeles, †Columbia University §{wasiahmad, kwchang}@cs.ucla.edu, †{saikatc, rayb}@cs.columbia.edu Abstract Code summarization and generation empower conversion between programming language (PL) and natural language (NL), while code translation avails the migration of legacy code from one PL to another. This paper introduces PLBART, a sequence-to-sequence model ca- pable of performing a broad spectrum of pro- gram and language understanding and gener- ation tasks. PLBART is pre-trained on an ex- tensive collection of Java and Python functions and associated NL text via denoising autoen- coding. Experiments on code summarization in the English language, code generation, and code translation in seven programming lan- guages show that PLBART outperforms or ri- vals state-of-the-art models. Moreover, exper- iments on discriminative tasks, e.g., program repair, clone detection, and vulnerable code de- tection, demonstrate PLBART’s effectiveness in program understanding. Furthermore, anal- ysis reveals that PLBART learns program syn- tax, style (e.g., identifier naming convention), logical flow (e.g., if block inside an else block is equivalent to else if block) that are crucial to program semantics and thus ex- cels even with limited annotations. 1 Introduction Engineers and developers write software programs in a programming language (PL) like Java, Python, etc., and often use natural language (NL) to com- municate with each other. Use of NL in software engineering ranges from writing documentation, commit messages, bug reports to seeking help in different forums (e.g., Stack Overflow), etc. Au- tomating different software engineering applica- tions, such as source code summarization, gener- ation, and translation, heavily rely on the under- standing of PL and NL—we collectively refer them as PLUG (stands for, Program and Language Un- derstanding and Generation) applications or tasks. ∗Equal contribution. Program snippet in Python 21 def sort_list(uns): return sorted(uns, key=lambda x:x[0]) Program snippet in Java 21 static Tuple[] sortArray(Tuple[] uns){ return Arrays.sort( 3 uns, new Comparator<Tuple>() { 4 public int compare( 5 Tuple o1, Tuple o2) { 6 return o1.get(0) == o2.get(0); 7 } 8 }); 9 } Summary: sort a list of tuples by first element Figure 1: Example motivating the need to understand the association of program and natural languages for code summarization, generation, and translation. Note that the use of NL in software development is quite different than colloquially written and spo- ken language. For example, NL in software de- velopment often contains domain-specific jargon, e.g., when software engineers use Code Smell1, it means a potential problem in code (something other than Smell in regular English language). In this work, our goal is to develop a general- purpose model that can be used in various PLUG applications. Recent advancements in deep learn- ing and the availability of large-scale PL and devel- opers’ NL data ushered in the automation of PLUG applications. One important aspect of …

Content truncated to 500 words