YNU-HPCC at ROCLING 2022 Shared Task: A Transformer-based Model with Focal Loss and Regularization Dropout for Chinese Healthcare Named Entity Recognition

YNU-HPCC at ROCLING 2022 Shared Task: A Transformer-based Model with Focal Loss and Regularization Dropout for Chinese Healthcare Named Entity Recognition Xiang Luo, Jin Wang and Xuejie Zhang School of Information Science and Engineering Yunnan University Kunming, China Contact: {wangjin,xjzhang}@ynu.edu.cn Abstract Named Entity Recognition (NER) is a fun- damental task in information extraction that locates the mentions of named en- tities and classifies them in unstructured texts. Previous studies typically used hid- den Markov model (HMM) and conditional random fields (CRF) for NER. To learn long-distance dependencies in text, recur- rent neural networks, e.g., LSTM and GRU can extract the semantic features for each token with a sequential manner. Based on Transformers, this paper describes the con- tribution to ROCLING-2022 Share Task. This paper adopts a transformer-based model with focal Loss and regularization dropout. The focal loss is to overcome the uneven distribution of the label. The reg- ularization dropout (r-drop) is to address the problem of vocabulary and descrip- tions that are too domain-specific. The ensemble learning is to improve the per- formance of the model. Comparative ex- periments were conducted on dev set to se- lect the model with the best performance for submission. That is, BERT model with BiLSTM-CRF, focal loss and R-Drop has achieved the best F1-score of 0.7768 and rank the 4th place. Keywords: Chinese Healthcare Named Entity Recognition, Sequence Labeling, Infor- mation Extraction, Transformers, Conditional Random Fields 1 Introduction Providing computer the ability to understand the abstract meaning of real world is a funda- mental task. The shared task of ROCLING- 2022 is Chinese healthcare named entity recog- nition task. Given a sentence about Chinese healthcare, the intelligent model is required to produce the entities in this sentence. Table 1 provides a detailed description of all target labels. For example, the input is 膽 汁長期滯留就會比較容易造成膽沙和膽結石 了。, the intelligence model is expected to ex- tract three entities, including 膽汁 as BODY, and both 膽沙 and 膽結石 as DISE. By using a sequence labeling approach, the correspond- ing labels for all tokens should be B-BODY, I-BODY, O, O, O, O, O, O, O, O, O, O, O, O, B-DISE, I-DISE, O, B-DISE, I-DISE, I-DISE, O, O. Here, the BIO schema is adopted, where B and I respectively means the begin and in- side labels, while O indicates that a token be- longs to other objects. Previous studies used probabilistic model for named entity recognition on text, such as hidden Markov model (HMM) (Zhou and Su, 2002) and conditional random field (CRF) (Zheng et al., 2017). Recent advances in deep neural networks (DNN) (Krizhevsky et al., 2012) and representation learning (Bengio et al., 2013) have considerably improved the ability of NER models. It mainly consists of an encoder to learn hidden representation for each token, as well as a classifier to assign a label for the token. For encoders, traditional models are usually used recurrent neural net- works (RNN), such as long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997) or gated recurrent units (GRU) to learn long- …

Content truncated to 500 words