YNU-HPCC at ROCLING 2023 MultiNER-Health Task: A transformer-based approach for Chinese healthcare NER

YNU-HPCC at ROCLING 2023 MultiNER-Health Task: A transformer-based approach for Chinese healthcare NER Chonglin Pang, You Zhang and Xiaobing Zhou School of Information Science and Engineering Yunnan University Kunming, China Contact: {yzhang0202, zhouxb}@ynu.edu.cn Abstract Chinese healthcare NER is an essential task in natural language processing to automatically identify healthcare entities such as symptoms, chemicals, diseases, and treatments for machine reading and understanding. Previous studies used Bi-directional Long Short-Term Memory (BiLSTM) and Conditional Random Fields (CRF) to solve NER tasks. This paper uses the RoBERTa-large pre-trained lan- guage model combined with BiLSTM-CRF to build a NER model suitable for Chinese healthcare tasks. Dropout is used to im- prove the performance and stability of the model, and gradient clipping is added to prevent gradient explosion. Comparative experiments were conducted on the dev set to select the model with the best perfor- mance for submission. The best model managed to achieve a macro-averaging F1 score of 68.40, which ranked second in the ROCLING 2023 shared task. Keywords: Chinese Healthcare Named Entity Recognition, RoBERTa, Bi-directional Long Short-Term Memory, Conditional Ran- dom Fields. 1 Introduction The shared task of ROCLING 2023 is MultiNER-Health Chinese Multi-genre Named Entity Recognition in the Healthcare Domain, which requires predicting the named entity boundaries and categories for each given sentence. The data sources for this task include Formal texts (FT), Social media (SM), and Wikipedia articles (WA), describing a total of ten types of entities related to Chinese healthcare. Table 1 provides the definitions and examples of entity types. The task uses the common BIO (Beginning, Inside, and Outside) format for NER. Here, the B-prefix before a tag indicates that the character is the beginning of a named entity, and the I-prefix before a tag indicates that the character is inside a named entity. An O tag indicates that a token belongs to no named entity (Lee and Lu, 2021). For example, the input is 早起 也能預防老化，甚至降低阿茲海默症的風險, the intelligence model is expected to extract two entities, including 老化 as SYMP, and 阿茲海默症 as DISE. Output according to BIO format as O, O, O, O, O, O, B-SYMP, I-SYMP, O, O, O, O, O, B -DISE, I-DISE, I-DISE, I-DISE, I-DISE, O, O, O. Early named entity recognition methods mainly used the Hidden Markov Model (HMM) (Zhou and Su, 2002) or Conditional Random Fields (Sutton et al., 2012) to train named entity recognition models on a large amount of manually labeled corpus. The model learns knowledge from a large amount of labeled corpus without the need for manu- ally defined rules. However, building a large- scale labeled corpus is time-consuming and la- borious. In recent years, deep learning algo- rithms have been applied in the field of nat- ural language processing (NLP). Named en- tity recognition methods based on deep learn- ing mainly include Recurrent neuralnetwork (RNN) and Long short-term memory (LSTM). Since LSTM solves the problem of gradient disappearance during long sequence training, BiLSTM-CRF became one of the mainstream models at that time. Once Transformer (Vaswani et al., 2017) emerged, it …

Content truncated to 500 words