On the Contribution of Per-ICD Attention Mechanisms to Classify Health Records in Languages with Fewer Resources than English

Proceedings of Recent Advances in Natural Language Processing, pages 165–172 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_020 165 On the Contribution of Per-ICD Attention Mechanisms to Classify Health Records in Languages With Fewer Resources than English Alberto Blanco1, Sonja Remmer2,3, Alicia P´erez1, Hercules Dalianis2,3, and Arantza Casillas1 1HiTZ Center - Ixa, University of the Basque Country UPV/EHU, Donostia, Spain 2Department of Computer and Systems Sciences, Stockholm University, Sweden 3Norwegian Centre for E-health Research, Tromsø, Norway {alberto.blanco,alicia.perez,arantza.casillas}@ehu.eus {remmer,hercules}@dsv.su.se Abstract We introduce a multi-label text classifier with per-label attention for the classification of Electronic Health Records according to the International Classification of Diseases. We apply the model on two Electronic Health Records datasets with Discharge Summaries in two languages with fewer resources than En- glish, Spanish and Swedish. Our model lever- ages the BERT Multilingual model (specifi- cally the Wikipedia, as the model have been trained with 104 languages, including Span- ish and Swedish, with the largest Wikipedia dumps1) to share the language modelling ca- pabilities across the languages. With the per- label attention, the model can compute the rel- evance of each word from the EHR towards the prediction of each label. For the experimental framework, we apply 157 labels from Chap- ter XI – Diseases of the Digestive System of the ICD, which makes the attention especially important as the model has to discriminate be- tween similar diseases. 1 Introduction Electronic Health Records (EHRs) are classified by clinical experts for documentation, reporting global health vital statistics, insurance billing, etc. International Classification of Diseases (ICD) is used world-wide to define diagnostic terms and procedures and serves to encode EHRs. There are thousands of terms encoded within the ICD WHO (2016). For medical experts, reading EHRs, lengthy and technical documents, finding explicit and implicit mentions of diagnoses and procedures for then assigning standard ICD codes is cumber- some and requires specific training. In fact, it is well-known that manual encoding is not error-free, as an example, Jacobsson and Serd´en (2013), es- timated that 20% of them were either incorrect or 1https://github.com/google- research/bert/blob/master/multilingual.md#list-of-languages were missing. In this context, natural language understanding brings opportunities to bridge the needs of the society in terms of computer aided coding approaches. In 2006, it was argued that Natural Language Processing (NLP) tools could quickly help iden- tify codes in discharge summaries Kukafka et al. (2006). Today NLP tools for classifying clinical documents written in English are widespread. Even more, languages with scarce resources for biomed- ical NLP like Spanish, Italian, Swedish, etc., are in the limelight in the last years to develop codi- fication systems as has been done for English. In the context of working towards the codification of documents, in languages with a small number of resources for NLP, different tasks have been addressed. In 2018 CLEF N´ev´eol et al. (2018b) worked with Italian, French and Hungarian for the automatic codification of death certificates. Each death certificate consisted of a few words (on aver- age 20 words) with at least one main diagnosis. In 2020 the CodiEsp task …

Content truncated to 500 words