Challenges of GPT-3-Based Conversational Agents for Healthcare

Proceedings of Recent Advances in Natural Language Processing, pages 619–630 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_067 619 Challenges of GPT-3-based Conversational Agents for Healthcare Fabian Lechner ‡⋆ and Allison Lahnala †‡ and Charles Welch † and Lucie Flek †‡ † Conversational AI and Social Analytics (CAISA) Lab Bonn-Aachen International Center for Information Technology (b-it), University of Bonn ‡ Department of Mathematics and Computer Science, University of Marburg ⋆ University Hospital Gießen and Marburg (UKGM) http://caisa-lab.github.io {fabian.lechner,allison.lahnala}@uni-marburg.de, {cwelch,lflek}@uni-bonn.de Abstract The potential to provide patients with faster information access while allowing medical spe- cialists to concentrate on critical tasks makes medical domain dialog agents appealing. How- ever, the integration of large-language mod- els (LLMs) into these agents presents certain limitations that may result in serious conse- quences. This paper investigates the chal- lenges and risks of using GPT-3-based mod- els for medical question-answering (MedQA). We perform several evaluations contextualized in terms of standard medical principles. We provide a procedure for manually designing pa- tient queries to stress-test high-risk limitations of LLMs in MedQA systems. Our analysis reveals that LLMs fail to respond adequately to these queries, generating erroneous medi- cal information, unsafe recommendations, and content that may be considered offensive. 1 Introduction There is growing interest in medical dialogue sys- tems that can support patients with health goals, extend access to health services such as informa- tion seeking, and improve the quality of patient care (Amith et al., 2020; Zeng et al., 2020). How- ever, there are many risks of patient-facing medical dialogue systems that could impose harms, such as the production of false or misleading information (Thirunavukarasu; Thirunavukarasu et al., 2023). In one study, for instance, Bickmore et al. (2018) found that the Google, Alexa, and Siri digital assis- tants answered 29% of medical questions in ways that could cause harm and 16% that could result in death. Further risks come with the use of large pre-trained language models (LLMs) in these sys- tems beyond inaccurate information that can yield negative conduct with the patients. Many works have highlighted ethical issues of LLMs, such as learned implicit social biases and generating offen- sive content, which are particularly concerning for medical contexts. Lin et al. (2022) find LLMs mem- orize abundant inaccurate medical information and popular misconceptions. Research and development of medical dialogue systems that employ LLMs typically evaluate the accuracy of the medical information and capabil- ities on medical tests. Medical soundness is only part of a comprehensive ethical evaluation of med- ical dialogue systems. It is imperative to further consider medical ethical principles and responsi- bilities that underlie the interpersonal nature of patient care and communication, which contribute to patient well-being (Zhou et al., 2023). In this study, we draw on standard ethical medical con- duct guidelines stated in the Medical Declaration of Geneva and principles of patient-centered ther- apy to develop an approach to target ethical risks in the evaluation of LLMs in medical applications. Our approach examines not only the risk of fac- tual hallucinations but also interpersonal, stylistic …

Content truncated to 500 words