Bio-Medical Multi-label Scientific Literature Classification using LWAN and Dual-attention module

Bio-Medical Multi-label Scientific Literature Classification using LWAN and Dual-attention module Deepanshu Khanna$, Aakash Bhatnagar%, Nidhir Bhavsar&, Muskaan Singh# and Petr Motlicek# $ ECED, Thapar Institute of Engineering and Technology, India % Boston University, Boston, Massachusetts &University of Potsdam, Potsdam, Germany # Speech and Audio Processing Group, IDIAP Research Institute, Martigny, Switzerland dkhanna be19@thapar.edu, aakash07@bu.edu, bhavsar@uni-potsdam.de, (msingh,petr.motlicek)@idiap.ch Abstract An enormous amount of research has been undertaken to overcome the severe impact of COVID-19 pandemic. These scientific findings are being reported in biomedical literature at a significant rate of 10,000 articles/month. In this paper, we tackle automated topic annotation for COVID-19 literature using SPECTER, Bioformer, and PubMedBERT embeddings using Label-Wise Attention Network (LWAN) based Multi-Label Document Classification (MLDC) using Dual- attention module. We also include literature from cardiovascular domain, to generalise our proposed approach. We significantly, achieve 87.71%,72.83% and 79.75% F1-score on LitCovid, Obsumed, WHO-Covid datasets. We release our code-base here https: //github.com/Deepanshu-beep/ MLDC_LWAN_Attention. 1 Introduction COVID-19 pandemic, has caused various unexpected challenges to public health and similar line of work. Since its outbreak, there has been a drastic loss in human life, leading to the exponential growth of re- search and innovation in this field , nearly 20000 articles ( till December 2019) have been published. Due to the information overload, it became a tremendous task for the general public and research professionals to keep up the pace with the latest COVID-19 research. The rate of increase in pub- lications related to the pandemic still continues to increase rapidly today. Most of the biomedical literature focuses on mul- tiple topics such as treatment, diagnosis, prevention, vaccine etc. These literature are often classified under multiple labels and therefore presents, Multi-Label Document Classification (MLDC) problem at a large scale. Previously, various researchers have performed thorough experimentation over MLDC. The task of MLDC has been covered in various applications es- pecially in medical domain. One of the most relevant datasets for the medical domain is the MIMIC-III [1], which contains an extensive literature of clinical notes for 16 ICD-9 codes. Similarly, LitCovid is an- other important dataset that can be used for MLDC, which contains various articles related to COVID-19 corresponding to their 7 unique labels. Traditional approaches for MLDC included ex- traction of handcrafted features from documents and then using single or multiple classifiers as in [2]. Ear- lier works in this domain used Convolutional Neu- ral Network (CNN) [3] and Seq-2-Seq [4]. Authors in [3], later extended their work in [5] by using a 1-dimensional convolutional network to learn text representations and evaluating their approach over 6 datasets. Another work [6] proved the effective- ness of systems involving attention mechanisms by combining Recurrent Neural Networks (RNNs) and self-attention network for MLDC. The paper [6] was one of the few works carrying out a comparison be- tween probabilistic label trees and neural models. Re- cent top-performing models include methodologies such as Transfer learning, Few and Zero-Shot learn- ing, and Label-Wise Attention Networks (LWANs). Articles [7, 8] experimented with transfer learning using pre-trained language …

Content truncated to 500 words