Exploring Data Augmentation for Code Generation Tasks

Findings of the Association for Computational Linguistics: EACL 2023, pages 1542–1550 May 2-6, 2023 ©2023 Association for Computational Linguistics Exploring Data Augmentation for Code Generation Tasks Pinzhen Chen* School of Informatics University of Edinburgh pinzhen.chen@ed.ac.uk Gerasimos Lampouras Noah’s Ark Lab Huawei gerasimos.lampouras@huawei.com Abstract Advances in natural language processing, such as transfer learning from pre-trained language models, have impacted how models are trained for programming language tasks too. Previous research primarily explored code pre-training and expanded it through multi-modality and multi-tasking, yet the data for downstream tasks remain modest in size. Focusing on data utilization for downstream tasks, we propose and adapt augmentation methods that yield consistent improvements in code translation and summarization by up to 6.9% and 7.5% re- spectively. Further analysis suggests that our methods work orthogonally and show benefits in output code style and numeric consistency. We also discuss test data imperfections. 1 Introduction Recent years have seen the rapid development of pre-trained models (PLMs) to enable knowledge transfer from generic texts to specific downstream tasks (Devlin et al., 2019; Liu et al., 2019). PLMs have been applied to the programming language do- main as well, following the same paradigm of (con- tinuing) training PLMs on code and text data, and then fine-tuning them for specific tasks (Kanade et al., 2020; Feng et al., 2020). PLMs are often adapted to programming languages by including code-specific modalities as part of the input like serialized syntax trees and data flows (Guo et al., 2021, 2022; Tipirneni et al., 2022). Such works have outperformed rule-based tools in various tasks, e.g. the CodeXGLUE benchmark (Lu et al., 2021). Despite the abundance of raw code available for pre-training, code data that meet downstream needs stay modest in size. This is due to the fact that, unlike texts, code datasets cannot be easily curated by people without programming knowledge. For *Work done during an internship at Huawei Noah’s Ark Lab. Our code will be available at https://github.com/huawei-noah/ noah-research/tree/master/NLP/DA4CodeGeneration example, code translation data in CodeXGLUE is sized at 10K, which is orders of magnitude smaller than their natural language counterparts that often include millions of instances (Kocmi et al., 2022). We are therefore motivated to enrich data in the fine-tuning phase of code PLMs, using auto- matic data augmentation (DA) methods like back- translation, monolingual, multilingual, and nu- meric augmentation. We extensively experiment on code translation, where a programming language is converted to another, and summarization, where a textual description is produced from a code block. Even with limited resources, we can lift perfor- mance by 6.9% for translation and 7.5% for sum- marization compared to baselines. Through man- ual inspection and extra evaluation measures, we demonstrate that our methods lead to desirable en- hancements special to code, namely better output code style and number correctness. 2 Methodology 2.1 Data synthesis Back-translation (BT, Sennrich et al., 2016) is a data augmentation technique originated from ma- chine translation, where an auxiliary model is used to construct pseudo-parallel data from monolingual resources. It can be straightforwardly applied …

Content truncated to 500 words