Transfer Learning for Named-Entity Recognition with Neural Networks

Transfer Learning for Named-Entity Recognition with Neural Networks Ji Young Lee∗1, Franck Dernoncourt∗1,2, Peter Szolovits1 1MIT, 2Adobe Research jjylee@mit.edu, dernonco@adobe.com, psz@mit.edu Abstract Recent approaches based on artificial neural networks (ANNs) have shown promising results for named-entity recognition (NER). In order to achieve high performances, ANNs need to be trained on a large labeled dataset. However, labels might be difficult to obtain for the dataset on which the user wants to perform NER: label scarcity is particularly pronounced for patient note de-identification, which is an instance of NER. In this work, we analyze to what extent transfer learning may address this issue. In particular, we demonstrate that transferring an ANN model trained on a large labeled dataset to another dataset with a limited number of labels improves upon the state-of-the-art results on two different datasets for patient note de-identification. Keywords: named-entity recognition, neural networks, transfer learning 1. Introduction Electronic health records (EHRs) have been widely adopted in some countries such as the United States and represent gold mines of information for medical research. The major- ity of EHR data exist in unstructured form such as patient notes (Murdoch and Detsky, 2013). Applying natural lan- guage processing on the unstructured data in conjunction with analyzing the other EHR data can lead to a better un- derstanding of health and diseases (Liao et al., 2015), and a more accurate phenotyping of patients to compare tests and treatments (Ananthakrishnan et al., 2013; Pivovarov and Elhadad, 2015; Halpern et al., 2016). However, before patient notes can be shared with medical investigators, some types of information, referred to as pro- tected health information (PHI), must be removed in order to preserve patient confidentiality. In the United States, the Health Insurance Portability and Accountability Act (HIPAA) (Office for Civil Rights, 2002) defines 18 different types of PHI, ranging from patient names and ID numbers to addresses and phone numbers. The task of removing PHI from a patient note is referred to as de-identification. The essence of de-identification is recognizing PHI in patient notes, which is a form of named-entity recognition (NER). Existing de-identification systems are often rule-based ap- proaches or feature-based machine learning approaches. However, these techniques require additional lead time for developing and fine-tuning the rules or features specific to each new dataset. Meanwhile, recent work using ANNs have yielded state-of-the-art performances without using any manual features (Dernoncourt et al., 2016). Compared to the previous systems, ANNs have a competitive advan- tage that the model can be fine-tuned on a new dataset with- out the overhead of manual feature development, as long as some labels for the dataset are available. However, it may still be inefficient to mass deploy ANN- based de-identification system in practical settings, since creating annotations for patient notes is especially difficult. ∗ These authors contributed equally to this work. This is due to the fact that only a restricted set of individ- uals is authorized to access original patient notes; the an- notation task cannot be crowd-sourced, making it slow and expensive …

Content truncated to 500 words