Comparative Analysis of Fine-tuned Deep Learning Language Models for ICD-10 Classification Task for Bulgarian Language

Proceedings of Recent Advances in Natural Language Processing, pages 1448–1454 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_162 1448 Comparative Analysis of Fine-tuned Deep Learning Language Models for ICD-10 classification task for Bulgarian Language Boris Velichkov1,2, Sylvia Vassileva1, Simeon Gerginov1, Boris Kraychev2, Ivaylo Ivanov1, Philip Ivanov1, Ivan Koychev1,2 and Svetla Boytcheva3 1 Faculty of Mathematics and Informatics, Sofia University ”St. Kliment Ohridski”, Sofia, Bulgaria sylvia.vassileva@gmail.com, simeongerginov1@gmail.com, sashovi@uni-sofia.bg, fsivanov@uni-sofia.bg, koychev@fmi.uni-sofia.bg 2 GATE Institute, Sofia University ”St. Kliment Ohridski”, Sofia, Bulgaria boris.velichkov@gate-ai.eu, boris.kraychev@gate-ai.eu 3 Institute of Information and Communication Technologies, Bulgarian Academy of Sciences, Sofia, Bulgaria svetla.boytcheva@iict.bas.bg Abstract The task of automatic diagnosis encoding into standard medical classifications and ontolo- gies is of great importance in medicine - both to support the daily tasks of physicians in the preparation and reporting of clinical documen- tation, and for automatic processing of clini- cal reports. In this paper, we investigate the application and performance of different deep learning transformers for automatic encoding in ICD-10 of clinical texts in Bulgarian. The comparative analysis attempts to find which approach is more efficient to be used for fine- tuning of pre-trained BERT family transformer to deal with a specific domain terminology on a rare language such as Bulgarian. On the one hand, we use SlavicBERT and Mul- tiligualBERT models, which are pre-trained for a common vocabulary in Bulgarian but lack medical terminology. On the other hand, we compare them to BioBERT, ClinicalBERT, SapBERT, BlueBERT models, which are pre- trained for medical terminology in English, but lack training for language models in Bulgar- ian, and vocabulary in Cyrillic. In our research study, all BERT models are fine-tuned with ad- ditional medical texts in Bulgarian and then applied to the classification task for encoding medical diagnoses in Bulgarian into ICD-10 codes. A big corpus of diagnoses in Bulgar- ian annotated with ICD-10 codes is used for the classification task. Such an analysis gives a good idea of which of the models would be suitable for tasks of a similar type and domain. The experiments and evaluation results show that both approaches have comparable accu- racy. 1 Introduction The task for automatic encoding of Electronic Health Records (EHR) with standard medical clas- sifications is a hot-topic. The international classifi- cation of diseases, 10th revision (ICD-10)1 is one of the most commonly used standard medical clas- sifications due to the availability of translations in several languages. It is a hierarchical classification that encodes each diagnosis into a standard code which is used for statistical analysis and insurance reimbursement. The current solutions for this task are based on a restricted subset of ICD-10 codes or address some specific task trained on a small manually annotated corpus. Recently some deep learning models like BERT (Devlin et al., 2018) pre-trained transformers were applied for different Natural Language Processing (NLP) and clinical NLP tasks. The first type of transformers is language models that cover com- mon vocabulary on several languages: SlavicBERT (Arkhipov et al., 2019) and MultiligualBERT (Pires et al., 2019). The second type is transformers that cover specific …

Content truncated to 500 words