NCUEE-NLP at SemEval-2023 Task 8: Identifying Medical Causal Claims and Extracting PIO Frames Using the Transformer Models

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 312–317 July 13-14, 2023 ©2023 Association for Computational Linguistics Abstract This study describes the model design of the NCUEE-NLP system for the SemEval- 2023 Task 8. We use the pre-trained transformer models and fine-tune the task datasets to identify medical causal claims and extract population, intervention, and outcome elements in a Reddit post when a claim is given. Our best system submission for the causal claim identification subtask achieved a F1-score of 70.15%. Our best submission for the PIO frame extraction subtask achieved F1-scores of 37.78% for Population class, 43.58% for Intervention class, and 30.67% for Outcome class, resulting in a macro-averaging F1-score of 37.34%. Our system evaluation results ranked second position among all participating teams. 1 Introduction A medical causal claim is an assertion that invokes causal relationships between variables such as “a drug has certain effects on a disease” (Pearl, 2004). A manually-annotated corpus for causal statements in PubMed publications has been used to train prediction models for identifying “correlational,” “conditional causal,” “direct causal,” and “no relationship” statements in research conclusion sections (Yu et al., 2019). BioBERT (Lee et al., 2020) has been shown to outperform BERT (Devlin et al., 2019) and linear SVM in identifying exaggerated causal claims made in health press releases (Yu et al., 2020). Empirical analyses have found that biomedical tweets are densely populated with claims (Wührl and Klinger, 2021). Entity- based claim representations have been proposed to extract condensed claims from medical tweets (Wührl and Klinger, 2022). The PICO framework is widely used to identify sentences in a given medical text belonging to four elements: 1) Population (P): this usually describes the characteristics of populations involved; 2) Intervention (I): this addresses the primary intervention along with any risk factors; 3) Comparison (C): this compares the efficacy of any new interventions with the primary intervention; and 4) Outcome (O): this measures the results of the intervention including improvements or side effects. The PICO elements have been used as a knowledge representation framework to analyze clinical questions (Huang et al., 2006). A corpus of clinical trial publications has been annotated with PICO elements (Zlabinger et al., 2018). PICO elements in medical texts have been detected based on long short-term memory networks (Jin and Szolovits, 2018). The contextualized BERT embedding has been used to enhance PIO element detection performance (Mezaoui et al., 2019). The SemEval-2023 Task 8 (Khetan, et al., 2023) organized a challenge to identify medical causal claims and extract related PIO frames. This task consists of two subtasks. 1) Subtask 1: for the provided snippet of text, participants should identify the span of text that belongs to either four defined categories, including “Claim,” “Experience,” “Experience based claim,” and “Question”. 2) Subtask 2: for the given text snippet and its identified claim, participants should extract the related “Population”, “Intervention,” and “Outcome” frames. This paper describes the NCUEE-NLP (National Central University, Dept. of Electrical NCUEE-NLP at SemEval-2023 Task 8: Identifying Medical Causal Claims and …

Content truncated to 500 words