SLEDGE-Z: A Zero-Shot Baseline for COVID-19 Literature Search

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4171–4179, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4171 SLEDGE-Z: A Zero-Shot Baseline for COVID-19 Literature Search Sean MacAvaney∗ † Arman Cohan‡ Nazli Goharian† †Information Retrieval Lab, Georgetown University, Washington DC ‡Allen Institute for AI, Seattle WA {sean,nazli}@ir.cs.georgetown.edu, armanc@allenai.org Abstract With worldwide concerns surrounding the Severe Acute Respiratory Syndrome Coron- avirus 2 (SARS-CoV-2), there is a rapidly growing body of scientific literature on the virus. Clinicians, researchers, and policy- makers need to be able to search these articles effectively. In this work, we present a zero- shot ranking algorithm that adapts to COVID- related scientific literature. Our approach fil- ters training data from another collection down to medical-related queries, uses a neural re- ranking model pre-trained on scientific text (SciBERT), and filters the target document col- lection. This approach ranks top among zero- shot methods on the TREC COVID Round 1 leaderboard, and exhibits a P@5 of 0.80 and an nDCG@10 of 0.68 when evaluated on both Round 1 and 2 judgments. Despite not relying on TREC-COVID data, our method outperforms models that do. As one of the first search methods to thoroughly evaluate COVID-19 search, we hope that this serves as a strong baseline and helps in the global crisis. 1 Introduction The emergence of the Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) prompted a worldwide research response. In the first 120 days of 2020, researchers published over 10,000 articles related to SARS-CoV-2 or COVID-19. Together with articles about similar viruses researched be- fore 2020, the body of research approaches 60,000 articles. Such a large body of research results in a considerable burden for those seeking informa- tion about various facets of the virus, including researchers, clinicians, and policy-makers. To help improve COVID-19 search, we intro- duce SLEDGE-Z: a simple yet effective zero-shot baseline for coronavirus Scientific knowLEDGE ∗ This work was done while at an internship at the Allen Institute for AI. What are the initial symptoms of COVID-19? COVID-19 subset Health Filter Date Filter Standard ranking pipeline Model Training Tune for Relevance Ranking Source Collection BM25 Re-Ranker SciBERT CORD-19 Query MS-MARCO Med-MARCO Ranked Results Figure 1: Overview of SLEDGE-Z. search. SLEDGE-Z adapts the successful BERT- based (Devlin et al., 2020) re-ranking model (Vanilla BERT, MacAvaney et al. (2019)) for COVID-19 search with three simple techniques. First, we propose a training data filtering tech- nique to help the ranking model learn relevance signals typical in medical text. The training data we use comes entirely from another dataset (MS- MARCO, Campos et al. (2016)), resulting in our model being zero-shot. Since MS-MARCO is a large collection of real user queries (over 800,000), it allows us to filter aggressively and still have ade- quate training data. Second, we replace the general contextualized language model BERT with one pre- trained on scientific literature (SciBERT, Beltagy et al. (2019)). This pre-training prepares the model for the type of language typically seen in scientific articles. Since the document collection …

Content truncated to 500 words