Code and Named Entity Recognition in StackOverflow

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4913–4926 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4913 Code and Named Entity Recognition in StackOverflow Jeniya Tabassum, Mounica Maddela, Wei Xu, Alan Ritter Department of Computer Science and Engineering The Ohio State University {bintejafar.1, maddela.4, xu.1265, ritter.1492}@osu.edu Abstract There is an increasing interest in studying nat- ural language and computer code together, as large corpora of programming texts become readily available on the Internet. For ex- ample, StackOverflow currently has over 15 million programming related questions writ- ten by 8.5 million users. Meanwhile, there is still a lack of fundamental NLP techniques for identifying code tokens or software-related named entities that appear within natural lan- guage sentences. In this paper, we introduce a new named entity recognition (NER) cor- pus for the computer programming domain, consisting of 15,372 sentences annotated with 20 fine-grained entity types. We trained in- domain BERT representations (BERTOver- flow) on 152 million sentences from Stack- Overflow, which lead to an absolute increase of +10 F1 score over off-the-shelf BERT. We also present the SoftNER model which achieves an overall 79.10 F1 score for code and named entity recognition on StackOver- flow data. Our SoftNER model incorporates a context-independent code token classifier with corpus-level features to improve the BERT- based tagging model.1 1 Introduction Recently there has been significant interest in modeling human language together with computer code (Quirk et al., 2015; Iyer et al., 2016; Yin and Neubig, 2018), as more data becomes available on websites such as StackOverflow and GitHub. This is an ambitious yet promising direction for scal- ing up language understanding to richer domains. Access to domain-specific NLP tools could help a wide range of downstream applications. For ex- ample, extracting software knowledge bases from 1Our code and data are available at: https:// github.com/jeniyat/StackOverflowNER/ Figure 1: Examples of software-related named entities in a StackOverflow post. text (Movshovitz-Attias and Cohen, 2015), devel- oping better quality measurements of StackOver- flow posts (Ravi et al., 2014), finding similar ques- tions (Amirreza Shirani, 2019) and more. How- ever, there is a lack of NLP resources and tech- niques for identifying software-related named en- tities (e.g., variable names or application names) within natural language texts. In this paper, we present a comprehensive study that investigates the unique challenges of named entity recognition in the social computer program- ming domain. These named entities are often ambiguous and have implicit reliance on the ac- companied code snippets. For example, the word ‘list’ commonly refers to a data structure, but can also be used as a variable name (Figure 1). In order to recognize these entities, we propose a software-related named entity recognizer (Soft- NER) that utilizes an attention network to com- bine the local sentence-level context with corpus- level information extracted from the code snip- pets. Using our newly annotated corpus of 15,372 sentences in StackOverflow, we rigorously test our proposed SoftNER model, which outperforms BiLSTM-CRF model and fine-tuned BERT model for identifying …

Content truncated to 500 words