Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5739 - 5754 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation Francesco Moramarco†‡, Alex Papadopoulos Korfiatis†, Mark Perera†, Damir Juric†, Jack Flann†, Ehud Reiter‡, Anya Belz‡, Aleksandar Savkov† †Babylon ‡University of Aberdeen † {francesco.moramarco, alex.papadopoulos, mark.perera, damir.juric, jack.flann, sasho.savkov}@babylonhealth.co.uk ‡ {r01fm20, ehud.reiter, anya.belz}@abdn.ac.uk Abstract In recent years, machine learning models have rapidly become better at generating clinical consultation notes; yet, there is little work on how to properly evaluate the generated consul- tation notes to understand the impact they may have on both the clinician using them and the patient’s clinical safety. To address this we present an extensive human evaluation study of consultation notes where 5 clinicians (i) listen to 57 mock consultations, (ii) write their own notes, (iii) post-edit a num- ber of automatically generated notes, and (iv) extract all the errors, both quantitative and qualitative. We then carry out a correlation study with 18 automatic quality metrics and the human judgements. We find that a sim- ple, character-based Levenshtein distance met- ric performs on par if not better than common model-based metrics like BertScore. All our findings and annotations are open-sourced. 1 Introduction Modern Electronic Health Records (EHR) systems require clinicians to keep a thorough record of every patient interaction and management deci- sion. While this creates valuable data that may lead to better health decisions, it also significantly increases the burden on the clinicians, with stud- ies showing this is a major contributor to burnout (Arndt et al., 2017). In most primary healthcare practices, the univer- sal record of a clinician-patient interaction is the SOAP (Subjective, Objective, Assessment, Plan) note, which captures the patient’s history, and the clinician’s observations, diagnosis, and manage- ment plan (Pearce et al., 2016). At the end of a consultation, the clinician is required to write up a SOAP note of the encounter. With the exception of the clinician’s internal observations on how the patient looks and feels, most of the SOAP note is verbalised and could be automatically constructed from the transcript of the consultation. A number of recent studies (Enarvi et al., 2020; Joshi et al., 2020; Zhang et al., 2021a) propose us- ing summarisation systems to automatically gener- ate consultation notes from the verbatim transcript of the consultation—a task henceforth referred to as Note Generation. Yet, there is very limited work on how to evaluate a Note Generation system so that it may be safely used in the clinical setting. Where evaluations are present, they are most often carried out with automatic metrics; while quick and cheap, these metrics were devised for general purpose summarisation or machine translation, and it is unclear whether they work just as well on this new task. In the field of automatic summarisa- tion and Natural Language Generation (NLG) in general, human evaluation is the gold standard pro- tocol. Even in cases where the cost of …

Content truncated to 500 words