COUGH: A Challenge Dataset and Models for COVID-19 FAQ Retrieval

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3759–3769 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3759 COUGH: A Challenge Dataset and Models for COVID-19 FAQ Retrieval Xinliang Frederick Zhang1,2,∗, Heming Sun1,3,*, Xiang Yue1, Simon Lin4, and Huan Sun1 1The Ohio State University 2University of Michigan 3University of Southern California 4Abigail Wexner Research Institute at Nationwide Children’s Hospital {zhang.9975,sun.2164,yue.149,sun.397}@osu.edu, Simon.Lin@nationwidechildrens.org Abstract We present a large, challenging dataset, COUGH, for COVID-19 FAQ retrieval. Simi- lar to a standard FAQ dataset, COUGH consists of three parts: FAQ Bank, Query Bank and Relevance Set. The FAQ Bank contains ∼16K FAQ items scraped from 55 credible websites (e.g., CDC and WHO). For evaluation, we in- troduce Query Bank and Relevance Set, where the former contains 1,236 human-paraphrased queries while the latter contains ∼32 human- annotated FAQ items for each query. We analyze COUGH by testing different FAQ re- trieval models built on top of BM25 and BERT, among which the best model achieves 48.8 under P@5, indicating a great challenge pre- sented by COUGH and encouraging future re- search for further improvement. Our COUGH dataset is available at https://github. com/sunlab-osu/covid-faq. 1 Introduction Many institutional websites today maintain an FAQ page to help users find relevant information for commonly asked questions. The FAQ retrieval task is defined as ranking FAQ items {(qi, ai)}1 from a collection given a user query Q (Karan and Šna- jder, 2016). In contrast to common Information Retrieval (IR), FAQ retrieval often introduces 3 new challenges: 1) brevity of FAQ texts in compar- ison with IR documents; 2) need for topic-specific knowledge; 3) usage of the new question field in FAQ items (Karan and Šnajder, 2016; Sakata et al., 2019). However, FAQ retrieval is under-studied compared with other IR applications such as open- domain QA (Chen and Yih, 2020). In this work, we specifically study FAQ retrieval for COVID-19, a contagious and fatal pandemic which is still evolving on a daily basis. Many web- sites like CDC and WHO provide quality informa- tion on COVID-19 and update FAQ pages regularly. ∗Work was done when the first two authors were at OSU. 1q and a are question and answer fields in an FAQ item. Question1:�Should�children�wear�masks? Answer1:�In�general,�children�2�years�and�older�should wear�a�mask...Appropriate�and�consistent�use�of�masks... FAQ�Bank Question2:�Coping�with�Self-Quarantine Answer2:�Remind�yourself�that�difficult�emotions�are normal�during�self-quarantine... Query1:�Is�it�possible�for�human�beings�to�get�sick�with COVID-19�transmitted�to�them�from�animals? Query2:�Is�it�possible�to�get�infected�by�COVID�19�if�I touch�food�surface�packaging? Query�Bank Question3:�COVID-19是如何在⼈与⼈之间传播的? (How�does�COVID-19�spread�between�people?) Answer3:�...该病毒的⼈际传播主要通过感染者与他⼈ 密切接触...(...mainly�when�an�infected�person�is�in�close contact�with�another�person...) Relevance�Set Query Relevant FAQ in FAQ Bank Score Query1 3.67 Query1 2.67 Q:�Can�wild�animals�spread�the�virus�that�causes�COVID-19�to people�or�pets?�A:�Currently,�there�is�no�evidence�to�suggest... Query2 3.67 Q:�How�is�COVID-19�transmitted?�A:�COVID-19�illness�is spread�mainly�from�person�to�person�through�respiratory... Q:�What�are�the�lab�protocols�for�identifying�the�virus�in�food? On�surfaces?A:�As�food�hasn't�been�implicated�in�transmission COUGH: The COVID-19 FAQ Dataset Figure 1: Examples from the COUGH dataset. To gain better insights into FAQ retrieval re- search and advance COVID-19 information search, we present an FAQ dataset, COUGH2, consisting of FAQ Bank, Query Bank and Relevance Set, follow- ing the standard of constructing an FAQ dataset (Manning et al., 2008). The FAQ Bank contains 15919 FAQ items scraped from 55 authoritative institutional websites (see a full list in Table A4 and A5). COUGH covers a wide range of topics on COVID-19, from general information about the virus to specific COVID-related instructions …

Content truncated to 500 words