How do we best approach the challenges of AI in medicine?
September 4, 2024
By Ethan Chupp, UNC Gillings School Communications Fellow
Generative artificial intelligence (AI) tools like ChatGPT have exploded in popularity in the past two years. These tools are already used in medicine and health care to assist with tasks humans find time-consuming. A new scoping review from Tara Templin, PhD, and Sean Sylvia, PhD, at the UNC Gillings School of Global Public Health, investigates the challenges presented by AI tools and how best to approach them.
Dr. Tara Templin
Dr. Sean Sylvia
Generative AI tools are “trained” on large datasets, such as text, images, video or specialized material. One type of generative AI technology, large language models (LLMs), are trained on text data— such as books, articles and websites—to generate novel text. Some LLM-based tools like OpenAI’s ChatGPT are designed to have a dialogue-based user interface for widespread use. Others, like Google’s Med-PaLM, are trained on medical data and are intended for specialized use.
Each system is designed to create content that looks or sounds like the information used for its training. Generative AI tools are powerful, but many questions remain about how they can be used safely in medical settings.
“These tools have a lot of potential to do a lot of good and to help on balance. I think they will be hugely beneficial for patients. But some risks need to be managed appropriately,” said Sylvia, associate professor in the Department of Health Policy and Management.
The review from Templin and Sylvia summarized 120 articles on medical and health-related uses of generative AI. The authors discuss six challenges that must be addressed.
Bias in generative AI models
Generative AI models often reflect biases present in their training data. These biases can lead to discriminatory or flawed medical recommendations. Techniques for reducing bias are sometimes employed, such as preprocessing data to remove discriminatory material. However, these methods are not foolproof and can introduce new biases from the users’ decisions. To combat this, many papers suggest that AI models be subjected to rigorous fairness evaluations and recommend more transparency in methods.
“Bias can creep in a million ways, in all stages of the process,” said Sylvia, “everything from the data itself to the data processing, to the way the AI system is used.”
Compromised data privacy
Generative AI models require substantial computational resources, often leading to reliance on third-party software for processing. This raises significant concerns about patient privacy, as sensitive medical data might be exposed to external entities. There may be fewer privacy concerns if institutions pursue local hosting of AI models, such as training an AI model with patient data from a single hospital.
While local hosting of AI models can reduce privacy risks, it requires significant infrastructure and expertise. Balancing these needs remains a challenge and not many papers suggested this strategy.
Misinterpretation of prompts
Generative AI can misunderstand user prompts, leading to inappropriate or inaccurate outputs. This is particularly concerning in medical contexts, where precise communication is crucial.
Additionally, these models are vulnerable to adversarial attacks, also known as jailbreaking, where specific sequences of characters can provoke harmful responses. Sometimes this can be as simple as prompting an LLM to ignore its safety guardrails. In a health care context, careful prompt writers can trick an AI tool into revealing its training data.
“Neither patients nor providers are thinking about jailbreaking. I don’t think people quite realize how possible it is to get a patient’s data from these systems. An adversarial attack like that could be quite harmful to data security and privacy,” said Templin, assistant professor of Health Policy and Management.
Hallucination
Generative AI models can produce factually incorrect outputs, known as “hallucinations.” In medicine, this could be a disaster if the information is used for a clinical decision or given directly to a patient.
“For patients, it’s probably not the best thing to go to ChatGPT and ask it for medical advice,” said Sylvia.
LLMs may be better suited for uses where hallucination is less of a worry, such as simplifying the reading level of approved medical advice. But this problem can be further reduced if experts review any information before it gets passed on to patients or the general public.
Overemphasis on language models
Templin and Sylvia found an overwhelming focus in the literature on LLMs like GPT-4. Researchers may be overlooking other types of AI systems that are more suitable for medical applications. Medical and public health fields should explore the full range of generative AI technologies, including specialized systems for medical imaging or drug discovery.
“Some companies have started bringing drugs to trial that have been developed by AI,” said Templin. “It could improve the efficiency of drug development pretty significantly.”
Dynamic nature of AI systems
Generative AI systems can change and adapt over time based on their interactions and experiences. Unlike other forms of software, where a consistent input will generally produce a predictable output, generative AI systems can be much more difficult to understand. Government agencies like the Food and Drug Administration may struggle to figure out how to regulate a piece of software that is constantly changing. It will remain a challenge to ensure ongoing compliance with regulations.
“I’ve started to see a shift away from regulating a single AI tool to more of an audit system, to see if the models are producing what the regulators expect them to,” added Templin.
Generative AI tools are coming. The benefits for medicine are vast, but the risks must be addressed thoughtfully.
Contact the UNC Gillings School of Global Public Health communications team at sphcomm@unc.edu.
