NLP Scholar: A Dataset for Examining the State of NLP Research

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 868–877 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 868 NLP Scholar: A Dataset for Examining the State of NLP Research Saif M. Mohammad National Research Council Canada saif.mohammad@nrc-cnrc.gc.ca Abstract Google Scholar is the largest web search engine for academic literature that also provides access to rich metadata associated with the papers. The ACL Anthology (AA) is the largest repository of articles on Natural Language Processing (NLP). We extracted information from AA for about 44 thousand NLP papers and identified authors who published at least three papers in AA. We then extracted citation information from Google Scholar for all their papers (not just their AA papers). This resulted in a dataset of 1.1 million papers and associated Google Scholar information. We aligned the information in the AA and Google Scholar datasets to create the NLP Scholar Dataset—a single unified source of information (from both AA and Google Scholar) for tens of thousands of NLP papers. NLP Scholar can be used to identify broad trends in productivity, focus, and impact of NLP research. We present here initial work on analyzing the volume of research in NLP over the years and identifying the most cited papers in NLP. We also list a number of additional potential applications. Keywords: Scientometrics, Trends in Research, Google Scholar, ACL Anthology, Citations 1. Introduction Google Scholar is a free web search engine for academic literature—peer reviewed journals, conferences, preprints, patents, theses, technical reports, etc.1 Through it, users can access the metadata associated with an article and of- ten the full text of the article as well. A key aspect of the metadata is the number of citations that an article has re- ceived. Google Scholar does not provide information on how many articles are included in its database. However, scientometric researchers have estimated that it included about 389 million documents in January 2018 (Gusenbauer, 2019)—making it the world’s largest source of academic information.2 Thus, it is not surprising that there is grow- ing interest in the use of Google Scholar information to draw inferences about scholarly research (Mart´ın-Mart´ın et al., 2018; Mingers and Leydesdorff, 2015; Ordu˜na-Malea et al., 2014; Howland, 2010). Our interests lie in the study of scholarly research in Natural Language Processing (NLP). However, Google Scholar does not provide information on the field of study pertaining to individual papers.3 Thus, in this project, we combine information from Google Scholar with a dedi- cated high-quality source of information for NLP papers, the ACL Anthology (AA). The ACL Anthology is a digital repository of public do- main, free to access, articles on Natural Language Process- ing (NLP).4 It includes papers published in the family of ACL conferences as well as in other NLP conferences such as LREC and RANLP.5 When it was first launched in 2002, 1https://scholar.google.com 2Scientometrics is the study of scientific literature using quan- titative techniques. 3Google Scholar does allow scholars to provide up to five tags …

Content truncated to 500 words