Examining Citations of Natural Language Processing Literature

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5199–5209 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5199 Examining Citations of Natural Language Processing Literature Saif M. Mohammad National Research Council Canada Ottawa, Canada saif.mohammad@nrc-cnrc.gc.ca. Abstract We extracted information from the ACL An- thology (AA) and Google Scholar (GS) to ex- amine trends in citations of NLP papers. We explore questions such as: how well cited are papers of different types (journal articles, con- ference papers, demo papers, etc.)? how well cited are papers from different areas of within NLP? etc. Notably, we show that only about 56% of the papers in AA are cited ten or more times. CL Journal has the most cited papers, but its citation dominance has lessened in re- cent years. On average, long papers get al- most three times as many citations as short pa- pers; and papers on sentiment classification, anaphora resolution, and entity recognition have the highest median citations. The anal- yses presented here, and the associated dataset of NLP papers mapped to citations, have a number of uses including: understanding how the field is growing and quantifying the impact of different types of papers. 1 Introduction The origins of Natural Language Processing (NLP) go back to the earliest work in Computer Science— when Alan Turing published his seminal paper ex- ploring whether machines can think, and proposed what is now known as the Turing test (Turing, 1950, 2009). A crucial factor in the evolution of NLP as a field of study in its own right was the forma- tion of the Association for Computational Linguis- tics (ACL) in 1962, and the first ACL conference in 1965.1 Today NLP is a broad interdisciplinary field with a growing number of researchers from Computer Science, Linguistics, Information Sci- ence, Psychology, Social Sciences, Humanities, and more joining its ranks. 1One can make a distinction between NLP and Computa- tional Linguistics; however, for this work, we will consider them to be synonymous. Also, ACL was originally named the Association for Machine Translation and Computational Linguistics (AMTCL). It was changed to ACL in 1968. Organizations such as ACL, ELRA, and AFNLP publish peer-reviewed NLP papers that include both journal articles and conference proceedings. Historically, the need for a faster review process has made conference proceedings the dominant form of published research in Computer Science and NLP. With time, the conferences and the types of pa- pers they publish, have evolved. Some conferences, such as EMNLP and ACL, are highly competitive, while others, such as most workshops and LREC, deliberately choose to keep more generous accep- tance rates. The publications themselves can be of different types: journal articles, conference papers, short papers, system demonstration papers, shared task papers, workshop papers, etc. New ideas and paradigms have evolved: for example, the rise of statistical NLP in the 1990s and deep learning in the 2010s. With the dawn of a new decade and NLP research becoming more diverse and more popular than it …

Content truncated to 500 words