RARR: Researching and Revising What Language Models Say, Using Language Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 126–173 July 9-14, 2023 ©2023 Association for Computational Linguistics Natural Language to Code Generation in Interactive Data Science Notebooks Pengcheng Yin∗, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Alex Polozov, Charles Sutton Google Inc. Abstract Computational notebooks, such as Jupyter notebooks, are interactive computing environ- ments that are ubiquitous among data scien- tists to perform data wrangling and analytic tasks. To measure the performance of AI pair programmers that automatically synthe- size programs for those tasks given natural language (NL) intents from users, we build ARCADE, a benchmark of 1,078 code gener- ation problems using the pandas data analy- sis framework in data science notebooks. AR- CADE features multiple rounds of NL-to-code problems from the same notebook. It requires a model to understand rich multi-modal con- texts, such as existing notebook cells and their execution states as well as previous turns of interaction. To establish a strong baseline on this challenging task, we develop PACH- INCO, a 62B code language model (LM) for Python computational notebooks, which sig- nificantly outperforms public code LMs. Fi- nally, we explore few-shot prompting strate- gies to elicit better code with step-by-step de- composition and NL explanations, showing the potential to improve the diversity and ex- plainability of model predictions. ARCADE is publicly available at https://github.com/ google-research/arcade-nl2code/. 1 Introduction Data science is the process of extracting insights from data (Wang et al., 2021a), and has become an integral part of decision making and knowl- edge discovery (Donoho, 2017). Data scientists and machine learning (ML) practitioners often use computational notebooks, which are interactive en- vironments such as Jupyter notebooks (Kluyver et al., 2016) and Google Colab, in their work. Data scientists spend a significant amount of time on data wrangling tasks to process raw data into us- able forms (illustrated in Fig. 1), as well as ex- ploratory data analysis (EDA) to gain insights ∗Correspondence to pcyin@google.com [1] import pandas as pd df = pd.read_csv('dataset/Gamepass_Games_v1.csv') [2] def get_avg(x): try: return float(x[0]) , float(x[1]) except: return 0, 0 df['min'], df['max'] = zip(*df['TIME'].str.replace( ' hours','').str.split("-").apply(get_avg)) Extract min and max hours as two columns [3] df['ADDED'] = pd.to_datetime( df['ADDED'],format="%d %b %y",errors='coerce') [4] df['GAMERS']=df['GAMERS'].str.replace( ',',' ').astype(int) added_year=df[df['GAMERS'].idxmax()]['ADDED'].year In which year was the most played game added? [5] df[(df['ADDED'].dt.year== added_date.year) & (df['RATING']>4)].groupby( df["ADDED"].dt.month)['GAME'].count() For each month in that year, how many games that has a rating of more than four? [6] fallout=df[df['GAME'].str.contains('Fallout')] fallout.groupby(fallout['ADDED'].dt.year).get_group( 2021)['max'].mean() What is the average maximum completion time for all fallout games added in 2021? [7] pd.pivot_table(df, index=df['ADDED'].dt.year, ..., aggfunc=np.count_nonzero, fill_value='0').rename_axis( index='Year', columns='Month') What is the amount of games added in each year for each month? (show a table with index as years, columns as months and fill null values with 0) Figure 1: An example of a computational notebook adapted from our dataset, with examples of reading data (cell c1), data wrangling (c2, c3), and exploratory data analysis (c4 …

Content truncated to 500 words