On the Importance of Data Size in Probing Fine-tuned Models

Findings of the Association for Computational Linguistics: ACL 2022, pages 9 - 19 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Compilable Neural Code Generation with Compiler Feedback Xin Wang1⋆⋄, Yasheng Wang2⋆, Yao Wan4, Fei Mi2, Yitong Li2,3, Pingyi Zhou2, Jin Liu1�, Hao Wu5, Xin Jiang2, Qun Liu2 1School of Computer Science, Wuhan University, China 2Huawei Noah’s Ark Lab, 3Huawei Technologies Co., Ltd. 4School of Computer Sci. & Tech., Huazhong University of Science and Technology, China 5School of Information Science and Engineering, Yunnan University, China {xinwang0920, jinliu}@whu.edu.cn, wanyao@hust.edu.cn, haowu@ynu.edu.cn {wangyasheng, feimi2, liyitong3, zhoupingyi, Jiang.Xin, qun.liu}@huawei.com Abstract Automatically generating compilable programs with (or without) natural language descrip- tions has always been a touchstone problem for computational linguistics and automated software engineering. Existing deep-learning approaches model code generation as text gen- eration, either constrained by grammar struc- tures in decoder, or driven by pre-trained lan- guage models on large-scale code corpus (e.g., CodeGPT, PLBART, and CodeT5). However, few of them account for compilability of the generated programs. To improve compilability of the generated programs, this paper proposes COMPCODER, a three-stage pipeline utilizing compiler feedback for compilable code gener- ation, including language model fine-tuning, compilability reinforcement, and compilability discrimination. Comprehensive experiments on two code generation tasks demonstrate the ef- fectiveness of our proposed approach, improv- ing the success rate of compilation from 44.18 to 89.18 in code completion on average and from 70.3 to 96.2 in text-to-code generation, respectively, when comparing with the state-of- the-art CodeGPT. 1 Introduction Automated code generation (or program synthe- sis) has attracted much attention over the past few years (Lu et al., 2021), because of its potential to improve the productivity of developers, as well as to speed up the software development (Parvez et al., 2021; Wang et al., 2021). In the life cycle of software development, different types of code generation tasks are desired, including code com- pletion (Liu et al., 2020b,a), text-to-code gener- ation (Hashimoto et al., 2018), program transla- tion (Chen et al., 2018), and program repair (Ya- sunaga and Liang, 2021). ⋆ Equal contribution. ⋄ Work done while this author was an intern at Huawei Noah’s Ark Lab. � Correspondence author. LM (a) Language Model Fine-Tuning LM Compiler Feedback (b) Compilability Reinforcement LM (c) Compilability Discrimination ... LM Discrimminator Candidates Compiler Feedback Compiler Compiler Code Snippet 1 def func(x,y): 2 out = Code Snippet 1 def func(x,y): 2 out = x * y 3 return out Code Snippet 1 def func(x,y): 2 out = Completing Code Snippet 1 def func(x,y): 2 out = x * y 3 return out Code Snippet 1 def func(x,y): 2 out = x * y 3 return out <BOS> <EOS> Figure 1: An illustration of Python code completion by COMPCODER, utilizing the compiler feedback with three stages. Recently, much effort has been made to ad- vance the development of code generation (Li et al., 2018), using different logical forms of code, such as the abstract syntax tree (AST) (Kim et al., 2021; Yin and Neubig, 2017; Rabinovich et al., …

Content truncated to 500 words