Large Language Models Meet NL2Code: A Survey

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7443–7464 July 9-14, 2023 ©2023 Association for Computational Linguistics Large Language Models Meet NL2Code: A Survey Daoguang Zan1,2∗, Bei Chen3, Fengji Zhang3, Dianjie Lu4, Bingchao Wu1, Bei Guan5, Yongji Wang5, Jian-Guang Lou3 1Cooperative Innovation Center, Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences; 3Microsoft Research Asia; 4Shandong Normal University 5Integrative Innovation Center, Institute of Software, Chinese Academy of Sciences {daoguang@, bingchao2017@, guanbei@, ywang@itechs.}iscas.ac.cn; {beichen, v-fengjzhang, jlou}@microsoft.com; Ludianjie@sdnu.edu.cn Abstract The task of generating code from a natural lan- guage description, or NL2Code, is considered a pressing and significant challenge in code in- telligence. Thanks to the rapid development of pre-training techniques, surging large lan- guage models are being proposed for code, sparking the advances in NL2Code. To facil- itate further research and applications in this field, in this paper, we present a comprehen- sive survey of 27 existing large language mod- els for NL2Code, and also review benchmarks and metrics. We provide an intuitive compari- son of all existing models on the HumanEval benchmark. Through in-depth observation and analysis, we provide some insights and con- clude that the key factors contributing to the success of large language models for NL2Code are "Large Size, Premium Data, Expert Tun- ing". In addition, we discuss challenges and opportunities regarding the gap between mod- els and humans. We also create a website https://nl2code.github.io to track the lat- est progress through crowd-sourcing. To the best of our knowledge, this is the first survey of large language models for NL2Code, and we believe it will contribute to the ongoing devel- opment of the field. 1 Introduction Is it possible for novice programmers, even those without any programming experience, to create software simply by describing their requirements in natural language? This is a long-standing fasci- nating question, which poses challenges to research areas like software engineering, programming lan- guage, and artificial intelligence. Realizing this scenario would have an unprecedented impact on our lives, education, economy, and labour mar- ket, as it would change the centralized software development and operation paradigm. Due to its ∗ This work was done before October 2022 when the author, Daoguang Zan, was an intern at Microsoft Research Asia. promising and intriguing future, natural-language- to-code (NL2Code) has been proposed as a re- search task that has attracted widespread interest in both academia and industry, with the goal of gener- ating code from natural language descriptions. Early studies on NL2Code were mainly based on heuristic rules or expert systems, such as probabilis- tic grammar-based methods (Joshi and Rambow, 2003; Cohn et al., 2010; Allamanis and Sutton, 2014) and those focusing on domain-specific lan- guages (de Moura and Bjørner, 2008; Gulwani, 2010; Jha et al., 2010), which are inflexible and not scalable. Other studies utilized static lan- guage models, like n-gram (Nguyen et al., 2013; Raychev et al., 2014; Devanbu, 2012) and Hid- den Markov (Sutskever et al., 2008), which have sparse vector representations …

Content truncated to 500 words