Recommendations for Datasets for Source Code Summarization

Proceedings of NAACL-HLT 2019, pages 3931–3937 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3931 Recommendations for Datasets for Source Code Summarization Alex LeClair, Collin McMillan Department of Computer Science and Engineering University of Notre Dame {aleclair, cmc}@nd.edu Abstract Source Code Summarization is the task of writing short, natural language descriptions of source code. The main use for these de- scriptions is in software documentation e.g. the one-sentence Java method descriptions in JavaDocs. Code summarization is rapidly becoming a popular research problem, but progress is restrained due to a lack of suit- able datasets. In addition, a lack of commu- nity standards for creating datasets leads to confusing and unreproducible research results – we observe swings in performance of more than 33% due only to changes in dataset de- sign. In this paper, we make recommendations for these standards from experimental results. We release a dataset based on prior work of over 2.1m pairs of Java methods and one sen- tence method descriptions from over 28k Java projects. We describe the dataset and point out key differences from natural language data, to guide and support future researchers. 1 Introduction Source Code Summarization is the task of writ- ing short, natural language descriptions of source code (Eddy et al., 2013). The most common use for these descriptions is in software documen- tation, such as the summaries of Java methods in JavaDocs (Kramer, 1999). Automatic gener- ation of code summaries is a rapidly-expanding research area at the crossroads of Computational Linguistics and Software Engineering, as a grow- ing tally of new workshops and NSF-sponsored meetings have recognized (Cohen and Devanbu, 2018; Quirk, 2015). The reason, in a nutshell, is that the vast majority of code summarization tech- niques are adaptations of techniques originally de- signed to solve NLP problems. A major barrier to ongoing research is a lack of standardized datasets. In many NLP tasks such as Machine Translation there are large, curated datasets (e.g. Europarl (Koehn, 2018)) used by several research groups. The benefit of these stan- dardized datasets is twofold: First, scientists are able to evaluate new techniques using the same test conditions as older techniques. And second, the datasets tend to conform to community customs of best practice, which avoids errors during evalua- tion. These benefits are generally not yet available to code summarization researchers; while large, public code repositories do exist, most research projects must parse and process these repositories on their own, leading to significant differences on one project to another. The result is that research progress is slowed as reproducibilty of earlier re- sults is difficult. Inevitably, differences in dataset creation also occur that can mislead researchers and over or un- derstate the performance of some techniques. For example, a recent source code summarization pa- per reports achieving 25 BLEU when generating English descriptions of Java methods with an ex- isting technique (Gu et al., 2018), which is 5 points higher than the original paper reports (Iyer et al., 2016). The …

Content truncated to 500 words