RNSum: A Large-Scale Dataset for Automatic Release Note Generation via Commit Logs Summarization

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8718 - 8735 May 22-27, 2022 c⃝2022 Association for Computational Linguistics RNSum: A Large-Scale Dataset for Automatic Release Note Generation via Commit Logs Summarization Hisashi Kamezawa1 Noriki Nishida2 Nobuyuki Shimizu3 Takashi Miyazaki3 Hideki Nakayama1 1 The University of Tokyo 2 RIKEN Center for Advanced Intelligence Project 3 Yahoo Japan Corporation hisas@nlab.ci.i.u-tokyo.ac.jp noriki.nishida@riken.jp {nobushim, takmiyaz}@yahoo-corp.jp nakayama@ci.i.u-tokyo.ac.jp Abstract A release note is a technical document that de- scribes the latest changes to a software product and is crucial in open source software devel- opment. However, it still remains challenging to generate release notes automatically. In this paper, we present a new dataset called RNSum, which contains approximately 82,000 English release notes and the associated commit mes- sages derived from the online repositories in GitHub. Then, we propose classwise extractive- then-abstractive/abstractive summarization ap- proaches to this task, which can employ a mod- ern transformer-based seq2seq network like BART and can be applied to various reposi- tories without specific constraints. The exper- imental results on the RNSum dataset show that the proposed methods can generate less noisy release notes at higher coverage than the baselines. We also observe that there is a signif- icant gap in the coverage of essential informa- tion when compared to human references. Our dataset and the code are publicly available. 1 Introduction Recently, there has been considerable interest in applying natural language processing (NLP) tech- niques to support software development (Iyer et al., 2016; Yin and Neubig, 2018; Panthaplackel et al., 2020). One such task involves the automatic gener- ation of release notes. A release note is a technical document that describes the latest changes to a software product, which is necessary for software developers to adjust their codes accurately for using the updated software. Since release notes are time- consuming to write manually, several studies have been done to explore automatic release note gen- eration. Moreno et al. (2014) proposed ARENA, Figure 1: An example data in RNSum; this example is derived from the release of tag v2.6.4 in https: //github.com/vuejs/vue. an automatic release notes generator, which first extracts and summarizes the changes in the source code and then integrates them with the informa- tion provided by version-control and issue-tracking systems. Pokorný (2020) developed Glyph, which classifies commit messages into predefined release- note categories (e.g., Features, Bug Fixes) using pre-trained word embeddings and produces the cat- egorized commit messages as the final release note. Despite the progress reported in these previous studies, usable release note generators are far from realization. We attribute this difficulty mainly to 8718 two problems. First, the existing resources for auto- matic release note generation are scarce; for exam- ple, Glyph was trained on only 5,000 commit mes- sages, which is too little for obtaining a sufficiently generalized model. Second, the existing methods have limited applicability; for example, ARENA re- quires an issue tracker hosted on Jira, thus prevent- ing it from being used for most GitHub …

Content truncated to 500 words