Incorporating External Knowledge through Pre-training for Natural Language to Code Generation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6045–6052 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6045 Incorporating External Knowledge through Pre-training for Natural Language to Code Generation Frank F. Xu∗, Zhengbao Jiang∗, Pengcheng Yin, Bogdan Vasilescu, Graham Neubig Carnegie Mellon University {fangzhex,zhengbaj,pcyin,vasilescu,gneubig}@cs.cmu.edu Abstract Open-domain code generation aims to gener- ate code in a general-purpose programming language (such as Python) from natural language (NL) intents. Motivated by the intuition that developers usually retrieve resources on the web when writing code, we explore the effectiveness of incorpo- rating two varieties of external knowledge into NL-to-code generation: automatically mined NL-code pairs from the online pro- gramming QA forum StackOverflow and programming language API documentation. Our evaluations show that combining the two sources with data augmentation and retrieval-based data re-sampling improves the current state-of-the-art by up to 2.2% absolute BLEU score on the code generation testbed CoNaLa. The code and resources are available at https://github.com/ neulab/external-knowledge-codegen. 1 Introduction Semantic parsing, the task of generating machine executable meaning representations from natural language (NL) intents, has generally focused on limited domains (Zelle and Mooney, 1996; Debo- rah A. Dahl and Shriber, 1994), or domain-specific languages with a limited set of operators (Berant et al., 2013; Quirk et al., 2015; Dong and Lap- ata, 2016; Liang et al., 2017; Krishnamurthy et al., 2017; Zhong et al., 2017; Yu et al., 2018, 2019b,a). However, recently there has been a move towards applying semantic parsing to automatically gener- ating source code in general-purpose programming languages (Yin et al., 2018; Yao et al., 2018; Lin et al., 2018; Agashe et al., 2019; Yao et al., 2019). Prior work in this area (Xiao et al., 2016; Ling et al., 2016; Rabinovich et al., 2017; Yin and Neubig, 2017, 2018; Dong and Lapata, 2018; Suhr et al., ∗The first two authors contributed equally. Annotated pairs <code, NL> External Knowledge Resources: Pre-train Mined pairs from Parsed pairs from API docs Text-to-Code Gen. Model Noisy but real-use distributed Clean but uniformly distributed Re-sampling w/ Real Distribution Human Curated Data: Real Distribution Estimation Fine-tune Figure 1: Our approach: incorporating external knowl- edge by data re-sampling, pre-training and fine-tuning. 2018; Iyer et al., 2018; Yin and Neubig, 2019) used a variety of models, especially neural architectures, to achieve good performance. However, open-domain code generation for general-purpose languages like Python is chal- lenging. For example, given the intent to choose a random file from the directory contents of the C drive, ‘C:\\’, one would expect the Python code snippet random.choice(os.listdir(‘C:\\’)), that realizes the given intent. This would involve not just generating syntactically correct code, but also using (and potentially combining) calls to APIs and libraries that implement some of the desired functionality. As we show in § 3, current code gen- eration models still have difficulty generating the correct function calls with appropriate argument placement. For example, given the NL intent above, although the state-of-the-art model by Yin and Neubig (2018) that uses a transition-based method to generate …

Content truncated to 500 words