Structured prediction models for RNN based sequence labeling in clinical text

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 856–865, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Structured prediction models for RNN based sequence labeling in clinical text Abhyuday N Jagannatha1, Hong Yu1,2 1 University of Massachusetts, MA, USA 2 Bedford VAMC and CHOIR, MA, USA abhyuday@cs.umass.edu , hong.yu@umassmed.edu Abstract Sequence labeling is a widely used method for named entity recognition and information extraction from unstructured natural language data. In the clinical domain one major ap- plication of sequence labeling involves ex- traction of relevant entities such as medica- tion, indication, and side-effects from Elec- tronic Health Record Narratives. Sequence la- beling in this domain presents its own set of challenges and objectives. In this work we experiment with Conditional Random Field based structured learning models with Recur- rent Neural Networks. We extend the pre- viously studied CRF-LSTM model with ex- plicit modeling of pairwise potentials. We also propose an approximate version of skip-chain CRF inference with RNN potentials. We use these methods1 for structured prediction in or- der to improve the exact phrase detection of clinical entities. 1 Introduction Patient data collected by hospitals falls into two cat- egories, structured data and unstructured natural lan- guage texts. It has been shown that natural text clinical documents such as discharge summaries, progress notes, etc are rich sources of medically rel- evant information like adverse drug events, medica- tion prescriptions, diagnosis information etc. Infor- mation extracted from these natural text documents can be useful for a multitude of purposes ranging 1Code is available at https://github.com/abhyudaynj/LSTM- CRF-models from drug efficacy analysis to adverse effect surveil- lance. A widely used method for Information Extrac- tion from natural text documents involves treating the text as a sequence of tokens. This format al- lows sequence labeling algorithms to label the rel- evant information that should be extracted. Sev- eral sequence labeling algorithms such as Condi- tional Random Fields (CRFs), Hidden Markov Mod- els (HMMs), Neural Networks have been used for information extraction from unstructured text. CRFs and HMMs are probabilistic graphical models that have a rich history of Natural Language Process- ing (NLP) related applications. These methods try to jointly infer the most likely label sequence for a given sentence. Recently, Recurrent (RNN) or Convolutional Neural Network (CNN) models have increasingly been used for various NLP related tasks. These Neu- ral Networks by themselves however, do not treat sequence labeling as a structured prediction prob- lem. Different Neural Network models use dif- ferent methods to synthesize a context vector for each word. This context vector contains informa- tion about the current word and its neighboring con- tent. In the case of CNN, the neighbors comprise of words in the same filter size window, while in Bidirectional-RNNs (Bi-RNN) they contain the en- tire sentence. Graphical models and Neural Networks have their own strengths and weaknesses. While graphical models predict the entire label sequence jointly, they usually rely on special hand crafted features to pro- vide good results. Neural Networks …

Content truncated to 500 words