Injecting knowledge into language generation: a case study in auto-charting after-visit care instructions from medical dialogue

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2373–2390 July 9-14, 2023 ©2023 Association for Computational Linguistics Injecting knowledge into language generation: a case study in auto-charting after-visit care instructions from medical dialogue Maksim Eremeev∗ Elemental Cognition New York University eremeev@nyu.edu Ilya Valmianski AuxHealth Xavier Amatriain Curai Health Anitha Kannan Curai Health Abstract Factual correctness is often the limiting fac- tor in practical applications of natural lan- guage generation in high-stakes domains such as healthcare. An essential requirement for maintaining factuality is the ability to deal with rare tokens. This paper focuses on rare tokens that appear in both the source and the refer- ence sequences, and which, when missed dur- ing generation, decrease the factual correctness of the output text. For high-stake domains that are also knowledge-rich, we show how to use knowledge to (a) identify which rare tokens that appear in both source and reference are impor- tant and (b) uplift their conditional probability. We introduce the “utilization rate” that encodes knowledge and serves as a regularizer by max- imizing the marginal probability of selected tokens. We present a study in a knowledge- rich domain of healthcare, where we tackle the problem of generating after-visit care instruc- tions based on patient-doctor dialogues. We verify that, in our dataset, specific medical con- cepts with high utilization rates are underesti- mated by conventionally trained sequence-to- sequence models. We observe that correcting this with our approach to knowledge injection reduces the uncertainty of the model as well as improves factuality and coherence without negatively impacting fluency. 1 1 Introduction Recent advances in language modeling (c.f. Dong et al. (2021); Erdem et al. (2022) for survey) have enabled applications across multiple domains including educa- tion (Shen et al., 2021), jurisprudence (Bell et al., 2021), e-commerce (Zhang et al., 2020; Xiao et al., 2021), and healthcare (Valmianski et al., 2021; Compton et al., 2021; Alambo et al., 2022; Krishna et al., 2020). One of the central challenges in deploying these mod- els in-the-wild is that rare words tend to have underesti- mated conditional probability during generation (Luong ∗Work done while at Curai 1Code is available at https://github. com/curai/curai-research/tree/main/ careplan-charting. et al., 2014; Chintagunta et al., 2021; Holtzman et al., 2020). However, in high-stakes applications, many of these rare words are semantically important and need to be preserved. For example, some symptoms, dis- eases, and medications can be both rare and important (Mottaghi et al., 2020) (e.g. knowing that the patient is taking warfarin is extremely important, even if the word “warfarin” occurs infrequently). Prior approaches for handling rare word generation utilize a copy mechanism (See et al., 2017; Joshi et al., 2020; Xu et al., 2020; Choi et al., 2021). This facilitates copying from the source text using a probabilistic switch to decide if the next output token is generated or copied from the input (See et al., 2017). However, it doesn’t properly resolve the main challenge: not all rare tokens are important. Only …

Content truncated to 500 words