PyMT5: multi-mode translation of natural language and Python code with transformers

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9052–9065, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 9052 PYMT5: multi-mode translation of natural language and PYTHON code with transformers Colin B. Clement∗ Microsoft Cloud and AI coclemen@microsoft.com Dawn Drain Microsoft Cloud and AI dadrain@microsoft.com Jonathan Timcheck† Stanford University timcheck@stanford.edu Alexey Svyatkovskiy Microsoft Cloud and AI alsvyatk@microsoft.com Neel Sundaresan Microsoft Cloud and AI neels@microsoft.com Abstract Simultaneously modeling source code and natural language has many excit- ing applications in automated software development and understanding. Pur- suant to achieving such technology, we introduce PYMT5, the PYTHON method text-to-text transfer transformer, which is trained to translate between all pairs of PYTHON method feature combinations: a single model that can both predict whole methods from natural language documen- tation strings (docstrings) and summarize code into docstrings of any common style. We present an analysis and modeling ef- fort of a large-scale parallel corpus of 26 million PYTHON methods and 7.7 mil- lion method-docstring pairs, demonstrat- ing that for docstring and method gen- eration, PYMT5 outperforms similarly- sized auto-regressive language models (GPT2) which were English pre-trained or randomly initialized. On the CODE- SEARCHNET test set, our best model pre- dicts 92.1% syntactically correct method bodies, achieved a BLEU score of 8.59 for method generation and 16.3 for docstring ∗Corresponding author †Work done during a Microsoft internship generation (summarization), and achieved a ROUGE-L F-score of 24.8 for method generation and 36.7 for docstring genera- tion. 1 Introduction Software is a keystone of modern society, touching billions of people through services and devices daily. Writing and documenting the source code of this software are challeng- ing and labor-intensive tasks; software devel- opers need to repeatedly refer to online doc- umentation resources in order to understand existing code bases to make progress. Devel- oper productivity can be improved by the pres- ence of source code documentation and a de- velopment environment featuring intelligent, machine-learning-based code completion and analysis tools. Recent progress in natural language process- ing (NLP), especially encoder/decoder-based transformer models (Vaswani et al., 2017) and pre-training (Radford et al., 2018; Lewis et al., 2019), has led to state-of-the-art per- formance on language modeling, classifica- tion (Devlin et al., 2018), translation (Raffel et al., 2019), summarization (Liu and Lap- 9053 ata, 2019), grammar correction (Bryant et al., 2017), entity recognition, dialogue genera- tion (Budzianowski and Vuli´c, 2019), and more. Along with these quantitative advances have come deeper understanding of the learned hidden representations which power transform- ers (Kovaleva et al., 2019; Voita et al., 2019; Clark et al., 2019; Ethayarajh, 2019). While they are arguably not ‘natural,’ programming languages are increasingly becoming model- ing playgrounds for NLP modeling. Since these languages by definition have a gram- mar, syntax, and known relationships between entities, they offer enticing opportunities for an even deeper probing of NLP models and tasks. Beyond theoretical importance, many NLP tasks have practical utility in software development environments: language model- ing or generation can be used for code com- pletion (Raychev et …

Content truncated to 500 words