HEVS-TUW at SemEval-2023 Task 8: Ensemble of Language Models and Rule-based Classifiers for Claims Identification and PICO Extraction

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1776–1782 July 13-14, 2023 ©2023 Association for Computational Linguistics HEVS-TUW at SemEval-2023 Task 8: Ensemble of Language Models and Rule-based Classifiers for Claims Identification and PICO Extraction Anjani Dhrangadhariya1,2† Wojciech Kusa3† Henning Müller1,2 Allan Hanbury3 1University of Geneva, Geneva, Switzerland 2HES-SO Valais-Wallis, Sierre, Switzerland {anjani.dhrangadhariya,henning.mueller}@hevs.ch 3TU Wien, Vienna, Austria {wojciech.kusa,allan.hanbury}@tuwien.ac.at Abstract This paper describes the HEVS-TUW team sub- mission to the SemEval-2023 Task 8: Causal Claims. We participated in two subtasks: (1) causal claims detection and (2) PIO identifica- tion. For subtask 1, we experimented with an ensemble of weakly supervised question detec- tion and fine-tuned Transformer-based models. For subtask 2 of PIO frame extraction, we used a combination of deep representation learning and a rule-based approach. Our best model for subtask 1 ranks fourth with an F1-score of 65.77%. It shows moderate benefit from en- sembling models pre-trained on independent categories. The results for subtask 2 warrant further investigation for improvement. 1 Introduction Identification and verification of causal claims from unstructured text data is essential for various decision-making processes, particularly in health- care. The SemEval-2023 Task 8 (Khetan et al., 2023) aims to advance the state-of-the-art in this area by focusing on two subtasks: identification of causal claims and extraction of Population, In- tervention, and Outcome (PIO) entities. The first subtask involves identifying the span of text that contains one of the four entities: a causal claim, a personal experience, a personal experience based on a claim or a question. This can be done at the sentence level, but only a part of a sentence may be annotated with one of these categories. The second subtask involves extracting the PIO frame related to the identified causal claim in a text snip- pet. The model utilizes both word-level, including contextual information and character-level features capturing different aspects of the data. † Equal contribution. For subtask 1, our approach consisted of an en- semble of pre-trained language models with a rule- based question classifier. Apart from training multi- class classification models, with the assumption that personal experience based on a claim category can be treated both as an instance of a personal experience and a causal claim, language models were also fine-tuned as three independent binary classifiers. In subtask 2, we experimented with a system combining a deep learning entity extraction pipeline incorporating different textual features and followed by a rule-based approach to combine sep- arate PIO tokens predictions into a consensus pre- diction sequence. Our approach to subtask 1 fared well at rank 4, with F1-score 32.77% better than the last ranked ap- proach and 12.7% behind the approach ranked first. For subtask 2, the approach ranks second last on the leaderboard, and the system mainly struggles with identifying the population frame. These subtasks have potential applications in content moderation, insurance claim identification, and hypothesis gen- eration from clinical notes. The shared task will motivate further research in this direction and lead to the development of more effective …

Content truncated to 500 words