CoSQA: 20,000+ Web Queries for Code Search and Question Answering

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5690–5700 August 1–6, 2021. ©2021 Association for Computational Linguistics 5690 CoSQA: 20,000+ Web Queries for Code Search and Question Answering Junjie Huang1∗, Duyu Tang2, Linjun Shou3, Ming Gong3, Ke Xu1, Daxin Jiang3, Ming Zhou2, Nan Duan2 1Beihang University 2Microsoft Research Asia 3Microsoft STC Asia 1huangjunjie@buaa.edu.cn, kexu@nlsde.buaa.edu.cn 2,3{dutang,lisho,migon,djiang,nanduan}@microsoft.com Abstract Finding codes given natural language query is beneficial to the productivity of software de- velopers. Future progress towards better se- mantic matching between query and code re- quires richer supervised training resources. To remedy this, we introduce the CoSQA dataset. It includes 20,604 labels for pairs of natural language queries and codes, each annotated by at least 3 human annotators. We further intro- duce a contrastive learning method dubbed Co- CLR to enhance query-code matching, which works as a data augmenter to bring more arti- ficially generated training instances. We show that evaluated on CodeXGLUE with the same CodeBERT model, training on CoSQA im- proves the accuracy of code question answer- ing by 5.1%, and incorporating CoCLR brings a further improvement of 10.5%. 1. 1 Introduction With the growing population of software develop- ers, natural language code search, which improves the productivity of the development process via retrieving semantically relevant code given natural language queries, is increasingly important in both communities of software engineering and natural language processing (Allamanis et al., 2018; Liu et al., 2020a). The key challenge is how to effec- tively measure the semantic similarity between a natural language query and a code. There are recent attempts to utilize deep neu- ral networks (Gu et al., 2018; Wan et al., 2019; Feng et al., 2020), which embed query and code as dense vectors to perform semantic matching in a unified vector space. However, these models are ∗Work done during internship at Microsoft Research Asia. 1The CoSQA data and leaderboard are available at https://github.com/microsoft/CodeXGLUE/tree/main/Text- Code/NL-code-search-WebQuery. The code is available at https://github.com/Jun-jie-Huang/CoCLR python check if path is absolute path or relative path Query: Code: def is_relative_url(url): """simple method to determine if a url is relative or absolute""" if url.startswith("#"): return None if url.find("://") > 0 or url.startswith("//"): # either 'http(s)://...' or '//cdn...' and therefore absolute return False return True Label: 1 capitalize letters in string python Query: Code: def snake_to_camel(s: str) -> str: """ Convert string from snake case to camel case. """ fragments = s.split('_’) return fragments[0] + ''.join(x.title() for x in fragments[1:]) Label: 0 Example 2: Example 1: Figure 1: Two examples in CoSQA. A pair of a web query and a Python function with documentation is an- notated with “1” or “0”, representing whether the code answers the query or not. mostly trained on pseudo datasets in which a natu- ral language query is either the documentation of a function or a tedious question from Stack Overflow. Such pseudo queries do not reflect the distribu- tion of real user queries that are frequently issued in …

Content truncated to 500 words