RedHOT: A Corpus of Annotated Medical Questions, Experiences, and Claims on Social Media

Findings of the Association for Computational Linguistics: EACL 2023, pages 809–827 May 2-6, 2023 ©2023 Association for Computational Linguistics RedHOT: A Corpus of Annotated Medical Questions, Experiences, and Claims on Social Media Somin Wadhwa† Vivek Khetan3 Silvio Amir† Byron C. Wallace† Northeastern University† Accenture AI Labs3 {wadhwa.s,s.amir,b.wallace}@northeastern.edu vivek.a.khetan@accenture.com Abstract We present Reddit Health Online Talk (RedHOT), a corpus of 22,000 richly anno- tated social media posts from Reddit spanning 24 health conditions. Annotations include de- marcations of spans corresponding to medi- cal claims, personal experiences, and ques- tions. We collect additional granular anno- tations on identified claims. Specifically, we mark snippets that describe patient Populations, Interventions, and Outcomes (PIO elements) within these. Using this corpus, we introduce the task of retrieving trustworthy evidence rel- evant to a given claim made on social media. We propose a new method to automatically de- rive (noisy) supervision for this task which we use to train a dense retrieval model; this out- performs baseline models. Manual evaluation of retrieval results performed by medical doc- tors indicate that while our system performance is promising, there is considerable room for improvement. We release all annotations col- lected (and scripts to assemble the dataset), and all code necessary to reproduce the results in this paper at: https://sominw.com/redhot. 1 Introduction Social media platforms such as Reddit provide in- dividuals places to discuss (potentially rare) med- ical conditions that affect them. This allows peo- ple to communicate with others who share in their condition, exchanging information about symptom trajectories, personal experiences, and treatment options. Such communities can provide support (Biyani et al., 2014) and access to information about rare conditions which may otherwise be dif- ficult to find (Glenn, 2015). However, the largely unvetted nature of social media platforms make them vulnerable to mis and disinformation (Swire-Thompson and Lazer, 2019). An illustrative and timely example is the idea that consuming bleach might be a viable treatment for r/ibs r/Psychosis r/Costochondritis I just ordered Metamucil bc I read psyllium may be better for IBS-D. Or maybe the fiber is what is making me go more? Definitely produces more gas. Surprising I'm seeing research articles that ketamine doesn't increase psychosis risk or induce psychosis past the duration of the drug. I only took a brief look into it. Has anyone here had ketamine induced psychosis? What is r/psychosis experience with ketamine? Ive had costo for a while, usually comes and goes. Done all the heart / lung checks all clear. Ive just recovered covid and what I'm left with is chest pain / pressure. I mean it could be a costo flare up which makes sense, but also been reading about myocarditis after covid and I’m worried. Figure 1: Examples of health-related Reddit posts anno- tated for populations, interventions, and outcomes. COVID-19,1 which quickly gained traction on so- cial media. All misinformation can be dangerous, but medical misinformation poses unique risks to public health, especially as individuals increasingly turn to social media to inform personal health deci- sions (Nobles et al., …

Content truncated to 500 words