Characterizing the Value of Information in Medical Notes

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2062–2072 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2062 Characterizing the Value of Information in Medical Notes Chao-Chun Hsu1, Shantanu Karnwal2, Sendhil Mullainathan3, Ziad Obermeyer4, Chenhao Tan2 1 University of Chicago,2 University of Colorado Boulder 3 Chicago Booth School of Business, 4 University of California, Berkeley chaochunh@uchicago.edu {shantanu.karnwal,chenhao.tan}@colorado.edu sendhil@chicagobooth.edu, zobermeyer@berkeley.edu Abstract Machine learning models depend on the qual- ity of input data. As electronic health records are widely adopted, the amount of data in health care is growing, along with complaints about the quality of medical notes. We use two prediction tasks, readmission prediction and in-hospital mortality prediction, to character- ize the value of information in medical notes. We show that as a whole, medical notes only provide additional predictive power over struc- tured information in readmission prediction. We further propose a probing framework to se- lect parts of notes that enable more accurate predictions than using all notes, despite that the selected information leads to a distribution shift from the training data (“all notes”). Fi- nally, we demonstrate that models trained on the selected valuable information achieve even better predictive performance, with only 6.8% of all the tokens for readmission prediction. 1 Introduction As electronic health records (EHRs) are widely adopted in health care, medicine is increasingly an information science (Stead et al., 2011; Short- liffe, 2010; Krumholz, 2014): Obtaining and an- alyzing information is critical for the diagnosis, prognosis, treatment, and prevention of disease. Although EHRs may increase the accuracy of stor- ing structured information (e.g., lab results), there are growing complaints about unstructured med- ical notes (henceforth “notes”) (Gawande, 2018; Payne et al., 2015; Hartzband et al., 2008). These complaints can be grouped into two per- spectives: consumption and production. On the one hand, information overload poses a critical challenge on the consumption side. That is, the sheer amount of information makes it difficult to glean meaningful information from EHRs, includ- ing notes (Weir and Nebeker, 2007). On the other hand, from the perspective of pro- duction, for every hour spent on patient interac- tion, physicians have an added one-to-two hours finishing the progress notes and reviewing results among other things, without extra compensation (Patel et al., 2018). The additional work con- tributes to physician burnout, along with low- quality notes and even errors in the notes. Con- sequently, physicians tend to directly copy large volumes of patient data into notes, but may fail to record information only available through interac- tion with patients. For instance, they may miss the wheezing breath for the diagnosis of the chronic obstructive pulmonary disease, or fail to have en- gaging conversations for evaluating signs of de- pression (Zeng, 2016). While the NLP community has focused on al- leviating the challenges in analyzing information (e.g., information overload), we argue that it is equally important to help caregivers obtain and record valuable information in the first place. We aim to take a first step towards this direction by characterizing …

Content truncated to 500 words