Incorporating Domain Knowledge into Language Transformers for Multi-Label Classification of Chinese Medical Questions

1047

Authors

nan

Abstract

AbstractIn this paper, we propose a knowledge infusion mechanism to incorporate domain knowledge into language transformers. Weakly supervised data is regarded as the main source for knowledge acquisition. We pre-train the language models to capture masked knowledge of focuses and aspects and then fine-tune them to obtain better performance on the downstream tasks. Due to the lack of publicly available datasets for multi-label classification of Chinese medical questions, we crawled questions from medical question/answer forums and manually annotated them using eight predefined classes: persons and organizations, symptom, cause, examination, disease, information, ingredient, and treatment. Finally, a total of 1,814 questions with 2,340 labels. Each question contains an average of 1.29 labels. We used Baidu Medical Encyclopedia as the knowledge resource. Two transformers BERT and RoBERTa were implemented to compare performance on our constructed datasets. Experimental results showed that our proposed model with knowledge infusion mechanism can achieve better performance, no matter which evaluation metric including Macro F1, Micro F1, Weighted F1 or Subset Accuracy were considered.

Keywords

nan

Full Content

� � �� 1,814 �� 8 �� 2,340�� 1.29 �� BERT � RoBERTa �� Macro F1�Micro F1�Weighted F1 � Subset Accuracy �� Abstract In this paper, we propose a knowledge infusion mechanism to incorporate domain knowledge into language transformers. Weakly supervised data is regarded as the main source for knowledge acquisition. We pre-train the language models to capture masked knowledge of focuses and aspects and then fine-tune them to obtain better performance on the downstream tasks. Due to the lack of publicly available datasets for multi-label classification of Chinese medical questions, we crawled questions from medical question/answer forums and manually annotated them using eight predefined classes: persons and organizations, symptom, cause, examination, disease, information, ingredient, and treatment. Finally, a total of 1,814 questions with 2,340 labels. Each question contains an average of 1.29 labels. We used Baidu Medical Encyclopedia as the knowledge resource. Two transformers BERT and RoBERTa were implemented to compare performance on our constructed datasets. Experimental results showed that our proposed model with knowledge infusion mechanism can achieve better performance, no matter which evaluation metric including Macro F1, Micro F1, Weighted F1 or Subset Accuracy were considered. �� Keywords: text classification, domain knowledge extraction, pretrained language models, biomedical informatics. 1� �� (transformer) �� BioBERT (��)�� (Masked Language Modeling, MLM) �� Incorporating Domain Knowledge into Language Transformers for Multi-Label Classification of Chinese Medical Questions �� Po-Han Chen, �� Yu-Xiang Zeng, �� Lung-Hao Lee �� Department of Electrical Engineering, National Central University {108521106, 109521127}@ncu.edu.tw, lhlee@ee.ncu.edu.tw �� The 33rd Conference on Computational Linguistics and Speech Processing (ROCLING 2021) Taoyuan, Taiwan, October 15-16, 2021. The Association for Computational Linguistics and Chinese Language Processing 265 � � �� (��) �� (Question Answering, QA) �� (Knowledge Infusion, KI) �� 1,814 �� 8�� 2,340�� 1.29 � �� Macro F1 � Micro F1 � Weighted F1 � Subset Accuracy�� 2� �� Zhang et al. (��) �� ERNIE�� BERT �� Liu et al. (��) �� K-BERT �� soft- position embedding �� BERT �� (knowledge noises) �� BERT �� K-BERT�� BERT �� Lee et al. (��) �� BERT� �� WordPiece Tokenization �� (Out-Of- Vocabulary, OOV)�� (state-of-the-art, SOTA) �� Xiong et al. (��) �� (weakly supervised) �� Wang et al. (��) � � � � � � � � � (knowledge embedding) �� Kepler�� He et al. (��) �� BERT �� …

Content truncated to 500 words

Labeled Text

Publication Details

Paper ID:: 1047
Year:: 2021
Publication:: TIPSTER
Publication Type:: conf
DOI:: nan

Links

Paper URL PDF Publication URL