MDACE: MIMIC Documents Annotated with Code Evidence

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7534–7550 July 9-14, 2023 ©2023 Association for Computational Linguistics MDACE: MIMIC Documents Annotated with Code Evidence Hua Cheng1, Rana Jafari1, April Russell1, Russell Klopfer1, Edmond Lu1, Benjamin Striner1, Matthew R. Gormley2 13M Health Information Systems, 2Carnegie Mellon University {hcheng, rjafari, arussell3, rklopfer, elu3, bstriner}@mmm.com, mgormley@cs.cmu.edu Abstract We introduce a dataset for evidence/rationale extraction on an extreme multi-label classifica- tion task over long medical documents. One such task is Computer-Assisted Coding (CAC) which has improved significantly in recent years thanks to advances in machine learning technologies. However, simply predicting a set of final codes for a patient encounter is insuf- ficient, as CAC systems are required to pro- vide supporting textual evidence to justify the billing codes. A model able to produce accu- rate and reliable supporting evidence for each code would be a tremendous benefit. However, a human-annotated code evidence corpus is ex- tremely difficult to create because it requires specialized knowledge. In this paper, we in- troduce MDACE, the first publicly available code evidence dataset, which is built on a sub- set of the MIMIC-III (English) clinical records. The dataset – annotated by professional med- ical coders – consists of 302 Inpatient charts with 3,934 evidence spans and 52 Profee charts with 5,563 evidence spans. We implemented several evidence extraction methods based on the EffectiveCAN model (Liu et al., 2021) to establish baseline performance on this dataset. MDACE can be used to evaluate code evidence extraction methods for CAC systems, as well as the accuracy and interpretability of deep learn- ing models for multi-label classification. We believe that the release of MDACE will greatly improve the understanding and application of deep learning technologies for medical coding and document classification. 1 Introduction In extreme multi-label text classification (XMLTC) a document is assigned a small number of labels from an extremely large set of possible labels. This large label space poses a challenge for ma- chine learning (ML) which is compounded by the length of input seen in long-document classifica- tion. While there is a wide range of document clas- sification datasets, only a limited number of those contain rationales or evidence associated with the labels. Of those that do, none (as of writing) are in the extreme multi-label classification setting or apply to long-documents. We present a new dataset for evidence extraction on long documents in an extreme multi-label classification setting. We also provide benchmark results using established tech- niques using neural networks. Computer-Assisted Coding (CAC) is a real world XMLTC application that uses natural lan- guage processing (NLP) techniques to extract pro- cedure and diagnosis codes from the documenta- tion of patient encounters. MIMIC-III (Medical In- formation Mart for Intensive Care) (Johnson et al., 2016) is an open-access dataset comprised of hos- pital records associated with patients admitted to the critical care units of the Beth Israel Deaconess Medical Center. For each patient record/chart, the data related to billing includes diagnostic codes, …

Content truncated to 500 words