Benchmark Data and Evaluation Framework for Intent Discovery Around COVID-19 Vaccine Hesitancy

Findings of the Association for Computational Linguistics: EACL 2023, pages 1358–1370 May 2-6, 2023 ©2023 Association for Computational Linguistics Benchmark Data and Evaluation Framework for Intent Discovery Around COVID-19 Vaccine Hesitancy Shai Gretz1∗, Assaf Toledo1∗, Roni Friedman1 Dan Lahav1, Rose Weeks2, Naor Bar-Zeev2 João Sedoc3, Pooja Sangha2, Yoav Katz1, Noam Slonim1 1IBM Research; 2Johns Hopkins Bloomberg School of Public Health; 3New York University {avishaig,roni.friedman-melamed,katz,noams}@il.ibm.com {assaf.toledo,dan.lahav}@ibm.com, {rweeks,nbarzee1,psangha1}@jhu.edu {jsedoc}@stern.nyu.edu Abstract The COVID-19 pandemic has made a huge global impact and cost millions of lives. As COVID-19 vaccines were rolled out, they were quickly met with widespread hesitancy. To address the concerns of hesitant people, we launched VIRA, a public dialogue system aimed at addressing questions and concerns surrounding the COVID-19 vaccines. Here, we release VIRADialogs, a dataset of over 8k dialogues conducted by actual users with VIRA, providing a unique real-world conver- sational dataset. In light of rapid changes in users’ intents, due to updates in guidelines or in response to new information, we highlight the important task of intent discovery in this use-case. We introduce a novel automatic eval- uation framework for intent discovery, leverag- ing the existing intent classifier of VIRA. We use this framework to report baseline intent- discovery results over VIRADialogs, that high- light the difficulty of this task. 1 Introduction As COVID-19 vaccines became available in late 2020, they were met with widespread vaccine hes- itancy (Goldstein et al., 2015; Sallam, 2021), a phenomena recognized as a top global concern by the World Health Organization (WHO) in 2019. To address such hesitancy, one needs accurate, reliable, and up to date information, constantly available to the general public. In recent years, task-oriented Dialogue Systems (DSs) have become an integral part of our daily lives, covering domains such as banking, tourism, and government agencies (Androutsopoulou et al., 2019). Correspondingly, we introduced VIRA,1 the Vaccine Information Resource Assistant – an infor- mational DS that aims to engage with COVID-19 vaccination concerns and questions. VIRA is able ∗These authors equally contributed to this work. 1https://vaxchat.org to respond to 181 different concerns, accumulated over the course of the pandemic. VIRA responses were written and vetted by leading medical and public health experts, relying on up-to-date facts and guidelines. An example of a dialog conducted with VIRA is presented in Table 1. We refer to this list of concerns and questions as intents. This is a slightly modified definition of intents, which are usually general tasks or goals which the user tries to accomplish (Jurafsky and Martin, 2009). In VIRA the intents are phrased as complete sentences, e.g., “Is the vaccine safe?”, as opposed to a synthetic class name like “vac- cine_safe”, and the goal of the intent classifier is to predict the correct intent, to which VIRA responds to the user with a pre-defined message. Advancing DSs depends on the availability of conversational datasets on which models can be trained. In domains where fact-based informa- tion is a necessity, e.g., healthcare, curating such datasets is often challenging: users in a crowd- sourcing …

Content truncated to 500 words