SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Mining

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5882–5893 August 1–6, 2021. ©2021 Association for Computational Linguistics 5882 SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Mining Taolin Zhang1,2,3, Zerui Cai4, Chengyu Wang2, Minghui Qiu2 Bite Yang5, Xiaofeng He4,6∗ 1 School of Software Engineering, East China Normal University 2 Alibaba Group 3 Shanghai Key Laboratory of Trsustworthy Computing 4 School of Computer Science and Technology, East China Normal University 5 DXY 6 Shanghai Research Institute for Intelligent Autonomous Systems zhangtl0519@gmail.com, zrcai flow@126.com, yangbt@dxy.cn {chengyu.wcy, minghui.qmh}@alibaba-inc.com, hexf@cs.ecnu.edu.cn Abstract Recently, the performance of Pre-trained Lan- guage Models (PLMs) has been significantly improved by injecting knowledge facts to en- hance their abilities of language understanding. For medical domains, the background knowl- edge sources are especially useful, due to the massive medical terms and their complicated relations are difficult to understand in text. In this work, we introduce SMedBERT, a med- ical PLM trained on large-scale medical cor- pora, incorporating deep structured semantics knowledge from neighbours of linked-entity. In SMedBERT, the mention-neighbour hybrid attention is proposed to learn heterogeneous- entity information, which infuses the semantic representations of entity types into the homo- geneous neighbouring entity structure. Apart from knowledge integration as external fea- tures, we propose to employ the neighbors of linked-entities in the knowledge graph as addi- tional global contexts of text mentions, allow- ing them to communicate via shared neighbors, thus enrich their semantic representations. Ex- periments demonstrate that SMedBERT signif- icantly outperforms strong baselines in various knowledge-intensive Chinese medical tasks. It also improves the performance of other tasks such as question answering, question matching and natural language inference.1 1 Introduction Pre-trained Language Models (PLMs) learn effec- tive context representations with self-supervised tasks, spotlighting in various NLP tasks (Wang et al., 2019a; Nan et al., 2020; Liu et al., 2020a). In addition, Knowledge-Enhanced PLMs (KEPLMs) (Zhang et al., 2019; Liu et al., 2020b; Wang et al., 2019b) further benefit language understanding by ∗Corresponding author. 1The code and pre-trained models will be available at https://github.com/MatNLP/SMedBERT. ��� ( sore throat) ������ (novel coronavirus) Symptom Cause of disease ���� (respiratory infection) Disease ����� (respiratory syndrome) Symptom-Disease Cause-Disease Entity Type Relation COVID-19 ������������ ��������������������(COVID-19)���� (Fever, sore throat, and diarrhea are symptoms of novel coronavirus (COVID-19).) ������������������������������ ������������� �� (pneumonia) ��� (infectious Department) Symptom-Symptom Medical Department �� (fever) Cause-Department Figure 1: Example of neighboring entity information in medical text. (Best viewed in color) grounding these PLMs with high-quality, human- curated knowledge facts, which are difficult to learn from raw texts. In the literatures, a majority of KEPLMs (Zhang et al., 2020a; Hayashi et al., 2020; Sun et al., 2020) inject information of entities corresponding to mention-spans from Knowledge Graphs (KGs) into contextual representations. However, those KEPLMs only utilize linked-entity in the KGs as auxiliary information, which pay little attention to the neighboring structured semantics informa- tion of the entity linked with text mentions. In the medical context, there …

Content truncated to 500 words