MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages

Findings of the Association for Computational Linguistics: EACL 2023, pages 265–273 May 2-6, 2023 ©2023 Association for Computational Linguistics MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages Zhiruo Wang∗♠ Grace Cuenca∗♦ Shuyan Zhou♠ Frank F. Xu♠ Graham Neubig♠♣ ♠Carnegie Mellon University ♦Princeton University ♣Inspired Cognition {zhiruow,shuyanzh,fangzhex,gneubig}@cs.cmu.edu, gcuenca@princeton.edu Abstract While there has been a recent burgeoning of ap- plications at the intersection of natural and pro- gramming languages, such as code generation and code summarization, these applications are usually English-centric. This creates a barrier for program developers who are not proficient in English. To mitigate this gap in technol- ogy development across languages, we propose a multilingual dataset, MCoNaLa, to bench- mark code generation from natural language commands extending beyond English. Mod- eled off of the methodology from the English Code/Natural Language Challenge (CoNaLa) dataset, we annotated a total of 896 NL-Code pairs in three languages: Spanish, Japanese, and Russian. We present a systematic evalu- ation on MCoNaLa by testing state-of-the-art code generation systems. Although the diffi- culties vary across three languages, all systems lag significantly behind their English counter- parts, revealing the challenges in adapting code generation to new languages.1 1 Introduction There are an increasing number of applications related to “code intelligence”, such as code sum- marization (Allamanis et al., 2016; Hu et al., 2018; Ahmad et al., 2020) and natural language (NL) to code generation (Ling et al., 2016; Rabinovich et al., 2017; Yin et al., 2018a; Xu et al., 2020; Norouzi et al., 2021; Wang et al., 2021), accompa- nied by code-specific tasks and benchmarks (Oda et al., 2015; Zhong et al., 2017; Yin et al., 2018b; Lu et al., 2021). However, in the cases where these benchmarks include natural language, that language is almost invariably English. There are a few exceptions, but most of them ei- ther focus on languages of specific domains (Sher- borne and Lapata, 2021; Sherborne et al., 2020; ∗Equal contribution. 1Code and data are available at https://github.com/zorazrw/ multilingual-conala Spanish ¿Cómo sumar el campo `precio` de todos los elementos del modelo `Precompra` en Django? (How to sum the `precio` field of all the elements of the `Precompra` model in Django?) totaldos = Precompra.objects.aggregate(Sum(precio)).values()[0]) Japanese 2次元配列`arr`の要素となっている1次元配列から先頭の値のみを抜き出す (Extract only the first value from the 1D array that is the element of the 2D array `arr`) arr[:, 0] Russian Установить кодировку `my_encode` для переменных окружения пользователя ‘username’ (Set `my_encode` encoding for ‘username’ environment variables) os.environ(‘username’).decode(my_encode) Figure 1: Examples in the MCoNaLa dataset, that aim to generate general-purpose Python code snippets from source intent of multiple natural languages. Moradshahi et al., 2020) or types of code (Oda et al., 2015; Liang et al., 2021), or contain NL in- tents collected via automatic translation (Li et al., 2021) (Appendix A). However, similarly to how Kwiatkowski et al. (2019) argue that “natural ques- tions” are necessary to appropriately benchmark QA systems, we argue that ensuring the naturalness and coverage of questions is essential for bench- marking code generation systems as well. A dataset for English code generation …

Content truncated to 500 words