Evaluating Retrieval for Multi-domain Scientific Publications

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4569–4576 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4569 Evaluating Retrieval for Multi-domain Scientific Publications Nancy Ide∗, Keith Suderman∗∗, Jingxuan Tu∗, Marc Verhagen∗, Shanan E. Peters‡, Ian Ross‡, John Lawson∗, Andrew Borg∗, James Pustejovsky∗ ∗Brandeis University, ∗∗Johns Hopkins University, ‡University of Wisconsin-Madison ∗Waltham, Massachusetts, USA, ∗∗Baltimore Maryland, USA, ‡Madison, Wisconsin, USA ide@cs.vassar.edu, suderman@jhu.edu, {jxtu, marc}@cs.brandeis.edu, peters@geology.wisc.edu, iross@cs.wisc.edu, lawsonj@gmail.com, borgchampa@gmail.com, jamesp@cs.brandeis.edu Abstract This paper provides an overview of the xDD/LAPPS Grid framework and provides results of evaluating the AskMe retrieval engine using datasets included in the BEIR benchmark. Our primary goal is to determine a solid baseline of performance to guide further development of our retrieval capabilities. Beyond this, we aim to dig deeper to determine when and why certain approaches perform well (or badly) on both in-domain and out-of-domain data, an issue that has to date received relatively little attention. Keywords: document retrieval, evaluation, retrieval benchmarks 1. Introduction Recently, retrieval and mining of scientific publications has emerged as an area of growing interest, in part due to the need to rapidly and effectively exploit the in- creasingly large body of COVID-19 literature for in- formation relevant to the current pandemic. Prior to the current crisis, scientists had already sought means to navigate the rapidly growing body of literature in domains beyond bioscience, which due to its exponen- tially increasing size has become unmanageable with- out automated assistance. The fundamental need is for efficient and accurate retrieval engines to locate articles relevant to a given topic, or which contain mentions of entities of interest to the researcher. Beyond this, scien- tists want to be able to extract specific data from mas- sive publication databases that cover a wide range of areas in a way that is accurate, repeatable, and leads to the discovery of new and related information. We are developing an end-to-end framework to ulti- mately accomplish this goal that builds on and inte- grates two major projects: (1) xDD1 (formerly known as GeoDeepDive) (Peters et al., 2017), one of the world’s largest and fastest-growing collections of sci- entific publications; and (2) the Language Applications (LAPPS) Grid2 (Ide et al., 2014), which provides an extensive suite of fully interoperable natural language processing (NLP) modules. A critical component of this framework is the AskMe retrieval engine (Suder- man et al., 2020)3, which serves as a bridge between the xDD publication database and the LAPPS Grid suite of NLP tools. AskMe’s fundamental function is to rapidly search xDD’s massive publication database in response to a user’s query, and subsequently pass all or some of the results directly into the LAPPS Grid for processing 1https://geodeepdive.org 2https://www.lappsgrid.org 3http://askme-ng-lapps.duckdns.org:8080/ask by NLP tools and predefined workflows in a one-stop, seamless operation. To that end, AskMe must be suit- ably robust to handle massive datasets and, especially, flexible enough to provide good results over documents from a wide range of scientific domains. Automated means to evaluate AskMe’s performance …

Content truncated to 500 words