SemEval-2023 Task 8: Causal Medical Claim Identification and Related PIO Frame Extraction from Social Media Posts

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2266–2274 July 13-14, 2023 ©2023 Association for Computational Linguistics SemEval-2023 Task 8: Causal Medical Claim Identification and Related PIO Frame Extraction from Social Media Posts Vivek Khetan Accenture Labs vivek.a.khetan@accenture.com Somin Wadhwa Northeastern University wadhwa.s@northeastern.edu Byron C. Wallace Northeastern University b.wallace@northeastern.edu Silvio Amir Northeastern University s.amir@northeastern.edu Abstract Identification of medical claims from user- generated text data is an onerous but essential step for various tasks including content mod- eration, and hypothesis generation. SemEval- 2023 Task 8 is an effort towards building those capabilities and motivating further research in this direction. This paper summarizes the de- tails and results of shared task 8 at SemEval- 2023 which involved identifying causal medi- cal claims and extracting related Populations, Interventions, and Outcomes (“PIO”) frames from social media (Reddit) text.1 This shared task comprised two subtasks: (1) Causal claim identification; and (2) PIO frame extraction. In total, seven teams participated in the task. Of the seven, six provided system descriptions which we summarize here. For the first subtask, the best approach yielded a macro-averaged F-1 score of 78.40, and for the second subtask, the best approach achieved token-level F-1 scores of 40.55 for Populations, 49.71 for Interven- tions, and 30.08 for Outcome frames. 1 Introduction Social media allows individuals to discuss, poten- tially rare, medical conditions with others and to find “long-tail” information about condition tra- jectories and treatment strategies (Wadhwa et al., 2023). While users may feel empowered, the unvetted nature of such online discourse makes it vulnerable to misinformation (factually incorrect statements, though not necessarily motivated by the aim to mislead) and disinformation (falsehoods crafted ex- plicitly to shape public opinion) (Swire-Thompson and Lazer, 2019). The problem of medical mis- information was particularly visible at the height of the COVID-19 pandemic, during which many unsubstantiated claims about treatments for the dis- ease circulated. For example, Ivermectin might be an effective treatment for COVID-19. 1https://causalclaims.github.io/ Here, we propose a task which enlists NLP to detect health related conversations on social media. This is a crucial first step for countering health mis- information (among other potential applications). More specifically, we propose a multi-stage task. The first objective is to identify causal medical claims (Khetan et al., 2022) made within a given text snippet from a Reddit post, consisting of ei- ther a single or multiple sentences, i.e., to identify the text spans that contains claims. The second objective is to extract key clinical elements from identified causal claims, in particular: The Popu- lation (i.e., the condition), the Intervention, and the Outcome i.e., the PIO elements (Richardson et al., 1995; Nye et al., 2018). For example, if we identify a claim such as “Drinking bleach can cure COVID-19!”, both the Population and Out- come would be “COVID-19”, while the Interven- tion would be “bleach”.2 In an identified claim, the extracted interventions and the outcomes have a causal relationship communicated between them, either explicitly or implicitly. This task generated a lot of interest from …

Content truncated to 500 words