CodeBERT: A Pre-Trained Model for Programming and Natural Languages
32340Authors
nan
Abstract
AbstractWe present CodeBERT, a bimodal pre-trained model for programming language (PL) and natural language (NL). CodeBERT learns general-purpose representations that support downstream NL-PL applications such as natural language code search, code documentation generation, etc. We develop CodeBERT with Transformer-based neural architecture, and train it with a hybrid objective function that incorporates the pre-training task of replaced token detection, which is to detect plausible alternatives sampled from generators. This enables us to utilize both “bimodal” data of NL-PL pairs and “unimodal data, where the former provides input tokens for model training while the latter helps to learn better generators. We evaluate CodeBERT on two NL-PL applications by fine-tuning model parameters. Results show that CodeBERT achieves state-of-the-art performance on both natural language code search and code documentation generation. Furthermore, to investigate what type of knowledge is learned in CodeBERT, we construct a dataset for NL-PL probing, and evaluate in a zero-shot setting where parameters of pre-trained models are fixed. Results show that CodeBERT performs better than previous pre-trained models on NLPL probing.
Keywords
Full Content
Findings of the Association for Computational Linguistics: EMNLP 2020, pages 137–147 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 137 Transformer-GCRF: Recovering Chinese Dropped Pronouns with General Conditional Random Fields Jingxuan Yang1, Kerui Xu1, Jun Xu2,3∗, Si Li1∗, Sheng Gao1, Jun Guo1, Ji-Rong Wen2,3 and Nianwen Xue4 1School of Artificial Intelligence, Beijing University of Posts and Telecommunications 2Gaoling School of Artificial Intelligence, Renmin University of China 3Beijing Key Laboratory of Big Data Management and Analysis Methods 4Department of Computer Science, Brandeis University {yjx, xukerui, lisi, gaosheng, guojun}@bupt.edu.cn junxu@ruc.edu.cn, jirong.wen@gmail.com, xuen@brandeis.edu Abstract Pronouns are often dropped in Chinese conver- sations and recovering the dropped pronouns is important for NLP applications such as Ma- chine Translation. Existing approaches usually formulate this as a sequence labeling task of predicting whether there is a dropped pronoun before each token and its type. Each utterance is considered to be a sequence and labeled in- dependently. Although these approaches have shown promise, labeling each utterance inde- pendently ignores the dependencies between pronouns in neighboring utterances. Model- ing these dependencies is critical to improving the performance of dropped pronoun recovery. In this paper, we present a novel framework that combines the strength of Transformer network with General Conditional Random Fields (GCRF) to model the dependencies be- tween pronouns in neighboring utterances. Re- sults on three Chinese conversation datasets show that the Transformer-GCRF model out- performs the state-of-the-art dropped pronoun recovery models. Exploratory analysis also demonstrates that the GCRF did help to cap- ture the dependencies between pronouns in neighboring utterances, thus contributes to the performance improvements. 1 Introduction In pro-drop languages such as Chinese, pronouns can be dropped as the identity of the pronoun can be inferred from the context, and this happens more frequently in conversations (Yang et al., 2015). Recovering dropped pronouns (DPs) is a critical task for many NLP applications such as Machine Translation where the dropped pronouns need to be translated explicitly in the target language (Wang et al., 2016a,b, 2018). Recovering dropped pro- noun is different from traditional pronoun resolu- tion tasks (Zhao and Ng, 2007; Yin et al., 2017, ∗ Corresponding author A1: (你) 去 巴西 的 时候 需要 提供 回程 机票 吗 ？ Do (you) need to provide the return ticket when you go to Brazil? B1: (我) 需要 Yes, (I) do. B2: (我) 要 把 ⾏程单 和 邀请函 打印 出来 带着 。 A2: 电⼦ ⾏程单 可以 吗 ？ Is electronic itinerary OK? B3: (你) 最好 还是 打印 ⼀ 份 吧 ， 省得 麻烦 。 (You)’d better print one copy to save trouble. A3: (previous utterance) 是的 Fine. A4: (我) 打印 两 份 吧 ， 张帆 是不是 也 需要 ？ (I) will print two copies. Does Fan Zhang also need it? B4: 我 打电话 问问 (他) I will ask (him) about it by phone. (I) need to print out the travel itinerary and invitation letter and bring them with me. Reply Expansion Acknowledge Figure 1: A conversation snippet between participant A and B. The dropped pronouns are shown …
Content truncated to 500 words
Labeled Text
AbstractWe present CodeBERT, a bimodal pre-trained model for programming language (PL) and natural language (NL). CodeBERT learns general-purpose representations that support downstream NL-PL applications such as natural language code search, code documentation generation, etc. We develop CodeBERT with Transformer-based neural architecture, and train it with a hybrid objective function that incorporates the pre-training task of replaced token detection, which is to detect plausible alternatives sampled from generators. This enables us to utilize both “bimodal” data of NL-PL pairs and “unimodal data, where the former provides input tokens for model training while the latter helps to learn better generators. We evaluate CodeBERT on two NL-PL applications by fine-tuning model parameters. Results show that CodeBERT achieves state-of-the-art performance on both natural language code search and code documentation generation. Furthermore, to investigate what type of knowledge is learned in CodeBERT, we construct a dataset for NL-PL probing, and evaluate in a zero-shot setting where parameters of pre-trained models are fixed. Results show that CodeBERT performs better than previous pre-trained models on NLPL probing.
Publication Details
- Paper ID:
- 32340
- Year:
- 2020
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan