Forgotten Knowledge: Examining the Citational Amnesia in NLP

Forgotten Knowledge: Examining the Citational Amnesia in NLP Janvijay Singh∗♣, Mukund Rungta∗♣, Diyi Yang◊, Saif M. MohammadΦ ♣Georgia Institute of Technology, ◊Stanford University, ΦNational Research Council Canada {iamjanvijay, mrungta8}@gatech.edu, diyiy@cs.stanford.edu, saif.mohammad@nrc-cnrc.gc.ca Abstract Citing papers is the primary method through which modern scientific writing discusses and builds on past work. Collectively, citing a di- verse set of papers (in time and area of study) is an indicator of how widely the community is reading. Yet there is little work looking at broad temporal patterns of citation. This work, systematically and empirically examines: How far back in time do we tend to go to cite pa- pers? How has that changed over time, and what factors correlate with this citational atten- tion/amnesia? We chose NLP as our domain of interest, and analyzed ∼71.5K papers to show and quantify several key trends in citation. No- tably, ∼62% of cited papers are from the imme- diate five years prior to publication, whereas only ∼17% are more than ten years old. Fur- thermore, we show that the median age and age diversity of cited papers was steadily increas- ing from 1990 to 2014, but since then the trend has reversed, and current NLP papers have an all-time low temporal citation diversity. Finally, we show that unlike the 1990s, the highly cited papers in the last decade were also papers with the least citation diversity; likely contributing to the intense (and arguably harmful) recency focus. Code, data, and a demo are available at the project homepage. 1 2 1 Introduction Study the past if you would define the future. — Confucius The goal of scientific research is to create a better future for humanity. To do this we innovate on ideas and knowledge from the past. Thus, a central characteristic of the scientific method and modern scientific writing is to discuss other work: to build on ideas, to critique or reject earlier conclusions, ∗Equal contribution. 1Code, data: https://github.com/iamjanvijay/ CitationalAmnesia/ 2Online demo: https://huggingface.co/spaces/ mrungta8/CitationalAmnesia/ to borrow ideas from other fields, and to situate the proposed work. Even when proposing something that others might consider dramatically novel, it is widely believed that these new ideas have been made possible because of a number of older ideas (Verstak et al., 2014). Citation (referring to another paper in a prescribed format) is the primary mech- anism to point the reader to these prior pieces of work and also to assign credit for shaping current work (Mohammad, 2020a; Rungta et al., 2022). Thus, we argue that examining citation patterns across time can lead to crucial insights into what we value, what we have forgotten, and what we should do in the future. Of particular interest is the extent to which good older work is being forgotten — citational amne- sia. More specifically, for this paper, we define citational amnesia as shown below: Citational Amnesia: the tendency to not cite enough relevant good work from the past (more than a few years old). We cannot directly measure citational amnesia empirically because determining …

Content truncated to 500 words