Python Code Generation by Asking Clarification Questions

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14287–14306 July 9-14, 2023 ©2023 Association for Computational Linguistics Python Code Generation by Asking Clarification Questions Haau-Sing Li1, Mohsen Mesgar2∗, André F. T. Martins3,4,5, Iryna Gurevych1 1Ubiquitous Knowledge Processing Lab (UKP Lab) Department of Computer Science and Hessian Center for AI (hessian.AI), TU Darmstadt 2Bosch Center for Artificial Intelligence, Renningen, Germany 3Instituto Superior Técnico and LUMLIS (Lisbon ELLIS Unit) 4Instituto de Telecomunicações, Lisbon, Portugal 5Unbabel hli@ukp.tu-darmstadt.de Abstract Code generation from text requires understand- ing the user’s intent from a natural language description and generating an executable code snippet that satisfies this intent. While re- cent pretrained language models demonstrate remarkable performance for this task, these models fail when the given natural language description is under-specified. In this work, we introduce a novel and more realistic setup for this task. We hypothesize that the under- specification of a natural language description can be resolved by asking clarification ques- tions. Therefore, we collect and introduce a new dataset named CodeClarQA containing pairs of natural language descriptions and code with created synthetic clarification questions and answers. The empirical results of our eval- uation of pretrained language model perfor- mance on code generation show that clarifi- cations result in more precisely generated code, as shown by the substantial improvement of model performance in all evaluation metrics. Alongside this, our task and dataset introduce new challenges to the community, including when and what clarification questions should be asked. Our code and dataset are available on GitHub.1 1 Introduction Text-to-code generation aims to understand a user’s intention represented by a natural language descrip- tion (NLD) to generate a code that satisfies the user’s intention. Models for this task are a crucial component of digital pair-programmers, which as- sist data scientists (Agashe et al., 2019; Liu et al., 2021), software developers (Chen et al., 2021; Xu et al., 2022), and computer programming educators (Li et al., 2022). Recent work addresses this task using pretrained language models (PLMs) fine-tuned on large-scale ∗ Work done while being a postdoc at UKP Lab. 1https://github.com/UKPLab/codeclarqa NLD: Logistic Regression. cls = LogisticRegression() grid_search = GridSearchCV(cls, [{'C':[0.1,1,10]}, {'penalty':['elasticnet'], 'C':[0.1,1,10], 'L1_ratio':np.linspace(0,1,6)}], scoring='accuracy',cv=4, return_train_score=True) grid_search.fit(X_train, y_train) model = grid_search.best_estimator_ joblib.dump(model, 'model.pkl') Do you want to call anything related to model/algorithm? If yes, which one? Yes, I want to call sklearn.GridSearchCV. Do you want to call joblib.dump documented as Serialize obj as bytes streamed into file? Yes. (a) sklearn.LogisticRegression sklearn.GridSearchCV sklearn.GridSearchCV.fit numpy.linspace joblib.dump sklearn.GridSearchCV.best_estimator_ (b) Figure 1: (a) An example of NLD-code pair that requires further clarification. We highlight operations that need clarification. (b) The generated graph of the example. Each node is an operation, with key operations marked in red and the rest in gray. Edges show the data flow. code data in general-purpose programming lan- guages, such as Python and Java (Chen et al., 2021; Li et al., 2022; Nijkamp et al., 2022; Chowdhery et al., 2022; Xu et al., 2022; Lahiri et al., 2022). Although these …

Content truncated to 500 words