Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5108–5120 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5108 Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports Yuhao Zhang1, Derek Merck2, Emily Bao Tsai1, Christopher D. Manning1, Curtis P. Langlotz1 1Stanford University 2University of Florida {yuhaozhang, ebtsai, manning, langlotz}@stanford.edu derek.merck@ufl.edu Abstract Neural abstractive summarization models are able to generate summaries which have high overlap with human references. However, ex- isting models are not optimized for factual correctness, a critical metric in real-world ap- plications. In this work, we develop a gen- eral framework where we evaluate the factual correctness of a generated summary by fact- checking it automatically against its reference using an information extraction module. We further propose a training strategy which op- timizes a neural summarization model with a factual correctness reward via reinforcement learning. We apply the proposed method to the summarization of radiology reports, where fac- tual correctness is a key requirement. On two separate datasets collected from hospitals, we show via both automatic and human evaluation that the proposed approach substantially im- proves the factual correctness and overall qual- ity of outputs over a competitive neural sum- marization system, producing radiology sum- maries that approach the quality of human- authored ones. 1 Introduction Neural abstractive summarization systems aim at generating sentences which compress a document while preserving the key facts in it (Nallapati et al., 2016b; See et al., 2017; Chen and Bansal, 2018). These systems are potentially useful in many real- world applications. For example, Zhang et al. (2018) have shown that customized neural abstrac- tive summarization models are able to generate radiology summary statements with high quality by summarizing textual findings written by radi- ologists. This task has significant clinical value because of its potential to accelerate the radiol- ogy workflow, reduce repetitive human labor, and improve clinical communications (Kahn Jr et al., 2009). Background: radiographic examination of the chest. clinical history: 80 years of age, male ... Findings: frontal radiograph of the chest demonstrates repositioning of the right atrial lead possibly into the ivc. ... a right apical pneumothorax can be seen from the image. moderate right and small left pleural effusions continue. no pulmonary edema is observed. heart size is upper limits of normal. Human Summary: pneumothorax is seen. bilateral pleural effusions continue. Summary A (ROUGE-L = 0.77): no pneumothorax is observed. bilateral pleural effu- sions continue. Summary B (ROUGE-L = 0.44): pneumothorax is observed on radiograph. bilateral pleu- ral effusions continue to be seen. Figure 1: A (truncated) radiology report and sum- maries with their ROUGE-L scores. Compared to the human summary, Summary A has high textual overlap (i.e., ROUGE-L) but makes a factual error; Summary B has a lower ROUGE-L score but is factually correct. However, while existing abstractive summariza- tion models are optimized to generate summaries that highly overlap with human references (Paulus et al., 2018), this does not guarantee factually cor- rect summaries, as shown in …

Content truncated to 500 words