Natural Language to Code Translation with Execution

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3533 - 3546 December 7-11, 2022 ©2022 Association for Computational Linguistics Natural Language to Code Translation with Execution Freda Shi1,2,∗ Daniel Fried1,3 Marjan Ghazvininejad1 Luke Zettlemoyer1,4 Sida I. Wang1 1Meta AI 2Toyota Technological Institute at Chicago 3Carnegie Mellon University 4University of Washington freda@ttic.edu dfried@cs.cmu.edu {ghazvini,lsz,sida}@fb.com Abstract Generative models of code, pretrained on large corpora of programs, have shown great success in translating natural language to code (Chen et al., 2021; Austin et al., 2021; Li et al., 2022, inter alia). While these models do not explic- itly incorporate program semantics (i.e., exe- cution results) during training, they are able to generate correct solutions for many problems. However, choosing a single correct program from a generated set for each problem remains challenging. In this work, we introduce execution result– based minimum Bayes risk decoding (MBR- EXEC) for program selection and show that it improves the few-shot performance of pre- trained code models on natural-language-to- code tasks. We select output programs from a generated candidate set by marginalizing over program implementations that share the same semantics. Because exact equivalence is in- tractable, we execute each program on a small number of test inputs to approximate seman- tic equivalence. Across datasets, execution or simulated execution significantly outperforms the methods that do not involve program se- mantics. We find that MBR-EXEC consistently improves over all execution-unaware selection methods, suggesting it as an effective approach for natural language to code translation.1 1 Introduction The recent success of large pretrained language models (Radford et al., 2019; Brown et al., 2020) has extended to translating natural language de- scriptions into executable code (Chen et al., 2021; Austin et al., 2021; Li et al., 2022, inter alia). After pretraining on large corpora of code with a simple language modeling objective, the models demon- strate the ability to follow few-shot prompts (Rad- ford et al., 2019; Brown et al., 2020) to translate nat- ∗Work done while interning at Meta AI. 1We open source our code at [this URL] and our data at [this URL]. 4 <info> assert add(1, 2) == 3 </info> <text> Add two integers </text> …<code> def add(a, b): return a + b </code> <info> assert count("abc1") == 3 </info> <text> Count lowercase letters </text> <code> def count(s): return len(s) </code> Codex def count(s): return len([c for c in s if c.islower()]) </code> def count(string): cnt = 0 for ch in string: if ch.islower(): cnt += 1 return cnt </code> count("abc1")= Execute 3 3 3 3 Code #3 Code #2 Code #1 1 1 0 0 1 1 Bayes Risk 1 1 2 Execution Result Matching Few-shot Prompt Test Example Figure 1: Illustration of MBR-EXEC on translating nat- ural language to Python code: we (1) sample programs from Codex (Chen et al., 2021), (2) execute each pro- gram on one test case, and (3) select the example with the minimal execution result–based Bayes risk. Num- bers around dotted lines denote the 0/1 matching loss …

Content truncated to 500 words