On the Embeddings of Variables in Recurrent Neural Networks for Source Code

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2679–2689 June 6–11, 2021. ©2021 Association for Computational Linguistics 2679 On the Embeddings of Variables in Recurrent Neural Networks for Source Code Nadezhda Chirkova HSE University∗ Moscow, Russia Abstract Source code processing heavily relies on the methods widely used in natural language pro- cessing (NLP), but involves specifics that need to be taken into account to achieve higher qual- ity. An example of this specificity is that the semantics of a variable is defined not only by its name but also by the contexts in which the variable occurs. In this work, we develop dynamic embeddings, a recurrent mechanism that adjusts the learned semantics of the vari- able when it obtains more information about the variable’s role in the program. We show that using the proposed dynamic embeddings significantly improves the performance of the recurrent neural network, in code completion and bug fixing tasks. 1 Introduction Deep learning is now actively being deployed in source code processing (SCP) for solving such tasks as code completion (Li et al., 2018), gen- erating code comments (Alon et al., 2019), and fixing errors in code (Vasic et al., 2019a). Source code visually looks like a text, motivating the wide use of NLP architectures in SCP. A lot of mod- ern SCP approaches are based on recurrent neural networks (Le et al., 2020), other popular architec- tures are transformers, and convolutional and graph neural networks. Utilizing the specifics of source code as a data domain may potentially improve the quality of neu- ral networks for SCP. These specifics include three main aspects. Firstly, the source code is strictly structured, i. e. the source code follows the syn- tactic rules of the programming language. Sec- ondly, the vocabularies may be large or even po- tentially unlimited, i. e. a programmer is allowed to define the identifiers of the arbitrary complex- ity. Thirdly, the identifiers are invariant to renam- ing, i. e. renaming all the user-defined identifiers ∗ The work was done while working at Samsung-HSE Laboratory, HSE University. does not change the algorithm that the code snip- pet implements. The first two mentioned specifics have been extensively investigated in the literature. For example, the tree-based architectures, such as TreeLSTM (Chen et al., 2018) or TreeTrans- former (Shiv and Quirk, 2019) allow for the uti- lization of the code’s structure. On the other hand, using byte-pair encoding (Karampatsis et al., 2020; Sennrich et al., 2016) or the anonymization of out- of-vocabulary identifiers (Chirkova and Troshin, 2021) deals with the unlimited vocabulary problem. However, the property of source code being in- variant to renaming user-defined identifiers has not been paid much attention to. In this work, we aim to close this gap for the recurrent neural networks (RNNs). Let us take a closer look at the invariance property. In Fig. 1 (a) and (b), a code snippet implementing a simple mathematical calculation is presented with two different …

Content truncated to 500 words