JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5436–5446, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5436 JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation Rajas Agashe, Srinivasan Iyer and Luke Zettlemoyer Paul G. Allen School of Computer Science and Engineering, Univ. of Washington, Seattle, WA {rajas, sviyer, lsz}@cs.washington.edu Abstract Interactive programming with interleaved code snippet cells and natural language mark- down is recently gaining popularity in the form of Jupyter notebooks, which accelerate pro- totyping and collaboration. To study code generation conditioned on a long context his- tory, we present JuICe, a corpus of 1.5 mil- lion examples with a curated test set of 3.7K instances based on online programming as- signments. Compared with existing contextual code generation datasets, JuICe provides re- fined human-curated data, open-domain code, and an order of magnitude more training data. Using JuICe, we train models for two tasks: (1) generation of the API call sequence in a code cell, and (2) full code cell generation, both conditioned on the NL-Code history up to a particular code cell. Experiments using current baseline code generation models show that both context and distant supervision aid in generation, and that the dataset is challenging for current systems. 1 Introduction Interactive computing (IC) is a software en- gidneering paradigm where programmers write source code scripts in an incremental fashion, one block at a time, taking decisions based on the out- put of execution of previously written blocks. An increasingly adopted platform for IC is the Jupyter notebook (Kluyver et al., 2016), which addition- ally encourages the use of markdown in natural language (NL) between code snippets as a means of documentation (Figure 1). Under the frame- work of IC, models that automatically generate fu- ture code blocks based on NL instructions, have the potential to provide significant assistance and speed up development. In this paper, we collect and release a new large-scale dataset based off of NL: Create and train the model. from sklearn.tree import DecisionTreeClassifier dtree = DecisionTreeClassifier() dtree.fit(X_train, y_train) from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split(X, y) d = 2 d = 1 Target import pandas as pd X = pd.read_json(‘features.json’) y = pd.read_json(‘labels.json’) NL: Split the data into train and test. d = 3 d = 4 NL: Load features and labels in a dataframe. d = 5 NL: Training a Decision Tree d = 6 Figure 1: This Python Jupyter notebook, comprising of interleaved NL markdown and code cells, loads data from a file and trains a decision tree classifier. We aim to generate the target code cell (blue) based on the pre- vious NL markdown Create and train the model and the prior NL and code history. To the left of each cell, d represents its distance from the target cell. Jupyter notebooks named JuICe (for Jupyter Inter- active Computing). We also present experiments to demonstrate baseline …

Content truncated to 500 words