The impact of lexical and grammatical processing on generating code from natural language

Findings of the Association for Computational Linguistics: ACL 2022, pages 2204 - 2214 May 22-27, 2022 c⃝2022 Association for Computational Linguistics The impact of lexical and grammatical processing on generating code from natural language Nathanaël Beau1,2 and Benoît Crabbé1 1 Université de Paris, LLF, CNRS, 75013 Paris, France 2 onepoint, 29 rue des Sablons, F-75116 Paris, France n.beau@groupeonepoint.com benoit.crabbe@u-paris.fr Abstract Considering the seq2seq architecture of Yin and Neubig (2018) for natural language to code translation, we identify four key compo- nents of importance: grammatical constraints, lexical preprocessing, input representations, and copy mechanisms. To study the impact of these components, we use a state-of-the-art architecture that relies on BERT encoder and a grammar-based decoder for which a formal- ization is provided. The paper highlights the importance of the lexical substitution compo- nent in the current natural language to code systems. 1 Introduction Translating natural language program descriptions to actual code is meant to help programmers to ease writing reliable code efficiently by means of a set of advanced code completion mechanisms. There are mainly two classes of methods for ob- taining code corresponding to a query expressed in natural language. The first one is code retrieval, which consists of searching and retrieving an ap- propriate code snippet from a code database. The second one is code generation, where the goal is to generate code fragments from a natural language description, generating potentially previously un- seen code. In this work, we are interested in Python code generation. Code generation features a mis- match between an ambiguous and noisy natural language input and the structured nature of the gen- erated code. Although Python’s vocabulary has a finite number of keywords, the set of values that can be assigned to a variable is infinite and constitutes one of the issues in predicting code corresponding to natural language. Like many other NLP tasks, current architectures for natural language to code generally take advan- tage of pre-trained language models such as BERT (Devlin et al., 2019) or GPT (Brown et al., 2020) based on the transformer architecture (Vaswani et al., 2017). In particular, these architectures are used for code generation where parallel data is limited due to the human expertise required for alignment. The best results on code generation are reached by pretraining seq2seq models on exter- nal sources, then by fine-tuning those models on smaller data sets. For instance, Orlanski and Git- tens (2021) fine-tune BART (Lewis et al., 2020) on data pairs of natural language and code and by taking advantage of external informations. Simi- larly, Norouzi et al. (2021) used BERT and a trans- former decoder in a semi-supervised way by taking advantage of a large amount of additional mono- lingual data. Another popular method is to train large language models on code (Austin et al., 2021; Hendrycks et al., 2021). Notably, GPT-3 has been finetuned on a large quantity of data from Github to obtain a powerful language model named Codex (Chen et al., 2021) that powers Github Copilot, a tool to …

Content truncated to 500 words