Geographic Citation Gaps in NLP Research

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1371 - 1383 December 7-11, 2022 ©2022 Association for Computational Linguistics Geographic Citation Gaps in NLP Research Mukund Rungta♢♣, Janvijay Singh♢♣, Saif M. Mohammad♡, Diyi Yang♢ ♢ School of Interactive Computing, Georgia Institute of Technology ♡ National Research Council Canada ♢ Stanford University {mrungta8, iamjanvijay}@gatech.edu saif.mohammad@nrc-cnrc.gc.ca diyiy@cs.stanford.edu Abstract In a fair world, people have equitable oppor- tunities to education, to conduct scientific re- search, to publish, and to get credit for their work, regardless of where they live. However, it is common knowledge among researchers that a vast number of papers accepted at top NLP venues come from a handful of western countries and (lately) China; whereas, very few papers from Africa and South America get pub- lished. Similar disparities are also believed to exist for paper citation counts. In the spirit of “what we do not measure, we cannot im- prove”, this work asks a series of questions on the relationship between geographical location and publication success (acceptance in top NLP venues and citation impact). We first created a dataset of 70,000 papers from the ACL An- thology, extracted their meta-information, and generated their citation network. We then show that not only are there substantial geograph- ical disparities in paper acceptance and cita- tion but also that these disparities persist even when controlling for a number of variables such as venue of publication and sub-field of NLP. Further, despite some steps taken by the NLP community to improve geographical diversity, we show that the disparity in publication met- rics across locations is still on an increasing trend since the early 2000s. We release our code and dataset here: https://github.com/ iamjanvijay/acl-cite-net. 1 Introduction Progress in science is accelerated by a sharing of ideas. However, there have been numerous in- stances in history where the predominance of one group of people in science, and the silencing of oth- ers, has led to the publication of harmful pseudo- science (Gould et al., 1996; Saini, 2019). Partic- ularly egregious examples include the publication of theories and ideas on racial hierarchy (Plutzer, ♣Equal contribution. 2013), male superiority (Huang et al., 2020), gen- der binary (Darwin, 2017), and eugenics (Cottrol, 2015). It has also been shown that a lack of in- clusion in invention and discovery leads to fewer technologies for the excluded group. For example, Koning et al. (2021) show how fewer technologies and health products are designed for women and Bender (2011), Bird (2020) and Mohammad (2019) show how a number of language technologies are designed for only a small number of languages. In this paper, we explore geographic inclusion in Natural Language Processing (NLP) research. Our premise is that in a fair world, people have equitable opportunities to education, to conduct scientific research, and to publish, regardless of where they live. However, researchers in the field know that a vast number of papers accepted at top NLP conferences and journals come from a handful of western countries and (lately) …

Content truncated to 500 words