Rethinking Positional Encoding in Tree Transformer for Code Representation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3204 - 3214 December 7-11, 2022 ©2022 Association for Computational Linguistics Rethinking Positional Encoding in Tree Transformer for Code Representation Han Peng, Ge Li∗, Yunfei Zhao, Zhi Jin∗ Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education; Institute of Software, EECS, Peking University, Beijing, China {phan, lige, zhaoyunfei, zhijin}@pku.edu.cn Abstract Transformers are now widely used in code rep- resentation, and several recent works further develop tree Transformers to capture the syn- tactic structure in source code. Specifically, novel tree positional encodings have been pro- posed to incorporate inductive bias into Trans- former. In this work, we propose a novel tree Transformer encoding node positions based on our new description method for tree struc- tures. Technically, local and global soft bias shown in previous works is both introduced as positional encodings of our Transformer model. Our model finally outperforms strong baselines on code summarization and comple- tion tasks across two languages, demonstrat- ing our model’s effectiveness. Besides, exten- sive experiments and ablation study shows that combining both local and global paradigms is still helpful in improving model performance. We release our code at https://github.com/ AwdHanPeng/TreeTransformer. 1 Introduction Machine learning for source code aims to learn the semantic embedding of programs. Due to the for- mat similarity between code and text (Hindle et al., 2016), Transformers (Vaswani et al., 2017) are now widely used in code representation (Hellendoorn et al., 2019; Zügner et al., 2021; Peng et al., 2021). Unlike natural language, source code is more logi- cal and has rich structures such as abstract syntax trees (AST). Therefore, one research topic of code intelligence is representing the syntax tree of code. Several recent works proposed novel tree-based Transformers by defining the position of each node to handle tree structure (Shiv and Quirk, 2019; Kim et al., 2020). In this work, we pursue the research line of tree Transformer for learning code AST. In Transformer, positional encoding is crucial to exploit potential structures of data (such as code ∗Corresponding authors ASTs or graphs) as other components are entirely position-invariant. Recently, a growing research trend is adapting Transformer to more complex structured data than plain text by modifying posi- tional encoding, not only in language processing (Wang et al., 2019b; Nguyen et al., 2020) but also in graph representation learning field (Dwivedi and Bresson, 2020; Mialon et al., 2021). As for the structure-aware Transformers, the key step of en- coding positions is to find a proper description for input structure, which means abstracting the physical structure of data into a suitable math- ematical form. For example, the vanilla Trans- former (Vaswani et al., 2017) regards the poten- tial text order in natural languages as the arrange- ment of natural numbers, while the graph Trans- formers (Kreuzer et al., 2021; Dwivedi and Bres- son, 2020) treat the positional relationship between graph nodes as the adjacent matrix or Laplacian fur- ther. Intuitively, a good description should be infor- mation …

Content truncated to 500 words