Automatic Classification of Evidence Based Medicine Using Transformers

Automatic Classification of Evidence Based Medicine Using Transformers Necva B¨ol¨uc¨u, Pınar Uskaner Hepsa˘g Computer Engineering Department Adana Alparslan T¨urkes¸ Science and Technology University Adana, Turkey {nbolucu, puskaner}@atu.edu.tr Abstract The goal of the shared task is multi-label classi- fication for biomedical records in English used for Evidence-Based Medicine. In this paper, we describe the model based on the Transformer submitted by our team turkNLP for the shared task. Our model achieved a Micro ROC score of ≈ 0.93 on the shared task and ranked 5th in the leaderboard. 1 Introduction The ALTA 2022 shared task1 is a well-studied Nat- ural Language Processing (NLP) problem which is multi-label sentence classification in biomedical field. The problem is assigning the sentences to one or more labels of the predefined 6 categories for the given dataset which is Evidence-Based Medicine (EBM) dataset presented by Kim et al. (2011). In this paper, we as a team of turkNLP have taken up and proposed a deep learning model based on Transformer (Vaswani et al., 2017) to identify the queries in Evidence-Based Medicine (EBM) pre- sented by Kim et al. (2011) for the ALTA 2022 shared task. Our model concatenates the encoder layer of the Transformer proposed by Vaswani et al (Vaswani et al., 2017) and the embedding of [CLS] token of the BERT model (Kenton and Toutanova, 2019), which is used as the embedding layer of the Transformer model. The main contribution of this paper is that we investigate the impact of the BERT model on the Transformer for multi-label classification problem. The model has shown an improvement over the Transformer model for multi-label classification, as the concatenation of the embedding of [CLS] token of the BERT model captures the semantic of the whole input, while the Transformer captures 1https://codalab.lisn.upsaclay.fr/ competitions/693500 the semantic of each word of the input2. The rest of the paper is organized as follows: We give the related work on the multi-label clas- sification problem for EBM with the shared task dataset Kim et al. (2011) in Section 2. The pro- posed model for the problem is given in Section 3 and the experimental setup, results, and detailed analysis of the results are presented in Section 4. Finally, Section 5 concludes the paper with insights on the impact of the proposed model on the multi- label classification problem for EBM and possible future work. 2 Related Work The first study classifying abstract sentences based on the PIBOSO scheme was conducted by Kim et al. (2011). The NICTA-PIBOSO dataset, the most studied dataset, was also published by Kim et al. (2011). The authors presented a Conditional Random Field (CRF) classifier with lexical (e.g., unigram, bigram), semantic (e.g., metathesaurus), structural (e.g., the position of the words), and se- quential (e.g., direct and indirect dependencies on previous sentences) features to assign sentences to predefined labels. Verbeke et al. (2012) presented a new approach based on kLog (Frasconi et al., 2014), a new lan- guage for statistical relational learning with kernels. In the study, the authors extracted …

Content truncated to 500 words