Staggered NLP-assisted refinement for Clinical Annotations of Chronic Disease Events

Staggered NLP-assisted refinement for Clinical Annotations of Chronic Disease Events Stephen T. Wu1,2, Chung-Il Wi1, Sunghwan Sohn1, Hongfang Liu1, Young J. Juhn1 1Mayo Clinic, Rochester, MN; 2Oregon Health & Science Univ., Portland, OR E-mail: wu.stephen@mayo.edu Abstract Domain-specific annotations for NLP are often centered on real-world applications of text, and incorrect annotations may be particularly unacceptable. In medical text, the process of manual chart review (of a patient's medical record) is error-prone due to its complexity. We propose a staggered NLP-assisted approach to the refinement of clinical annotations, an interactive process that allows initial human judgments to be verified or falsified by means of comparison with an improving NLP system. We show on our internal Asthma Timelines dataset that this approach improves the quality of the human-produced clinical annotations. Keywords: annotation refinement, clinical text, staggered approaches, rule-based NLP 1. Introduction Linguistic annotations for core Natural Language Processing (NLP) tasks – like POS tagging or parsing – are inherently analytic explanations of text. The purpose of a POS tag is to describe the grammatical structure of words in an utterance, and a corresponding resource aims to help computational methods reduce ambiguity in what is communicated by the text. In contrast, domain-specific annotations for NLP are often centered on real-world applications of text. A common reason to annotate events in medical or legal documentation is to highlight actionable information; this kind of language resource aims to help computational methods reduce disagreement in what is decided based on the text. In this work, we describe the construction of Asthma Timelines, a clinically actionable resource in which chronic disease symptoms and events are annotated over clinical text. More specifically, with the help of medical experts, we annotated the prototypical chronic disease of asthma with its natural history (timeline of events) and relevant symptoms. These annotations span over multiple documents per patient, and are intended to provide evidence for a patient-level medical decision: is the patient’s asthma status negative, positive, (in subsequent) remission, or (has it progressed to) relapse? We observe that the challenging clinical nature of this problem makes gold standard annotations difficult to construct. Patients’ asthma symptoms are primarily found in clinical text, which requires either NLP or human chart review to utilize; lab tests and other structured data fields are insufficient. While previous annotation projects (doing medical chart review) have reported reasonable agreement scores following a chart review guideline, the small amounts of associated data would be considered noisy training data for a machine learning algorithm. There are multiple clinical guidelines for what constitutes asthma (Yunginger et al. 1992a), and an annotator may not always follow one guideline rigorously. Also, while a human annotator can take context into account, there are times when this incorporation of contextual information can lead to inconsistent or even incorrect annotations. The typical means of overcoming noise in human annotations – using multiple annotators and measuring inter-annotator agreement – is a significant barrier when expert clinical annotators are necessary. These challenges are especially acute in privacy-protected domain-specific language …

Content truncated to 500 words