ChatGPT found by study to spread inaccuracies when answering medication questions

ChatGPT has been found to have shared inaccurate information regarding drug usage, according to new research.

In a study led by Long Island University (LIU) in Brooklyn, New York, nearly 75% of drug-related, pharmacist-reviewed responses from the generative AI chatbot were found to be incomplete or wrong.

In some cases, ChatGPT, which was developed by OpenAI in San Francisco and released in late 2022, provided "inaccurate responses that could endanger patients," the American Society of Health System Pharmacists (ASHP), headquartered in Bethesda, Maryland, stated in a press release.

WHAT IS ARTIFICIAL INTELLIGENCE?

ChatGPT also generated "fake citations" when asked to cite references to support some responses, the same study also found.

Along with her team, lead study author Sara Grossman, PharmD, associate professor of pharmacy practice at LIU, asked the AI chatbot real questions that were originally posed to LIU’s College of Pharmacy drug information service between 2022 and 2023.

ChatGPT, the AI chatbot created by OpenAI, generated inaccurate responses about medications, a new study has found. The company itself previously said that "OpenAI’s models are not fine-tuned to provide medical information. You should never use our models to provide diagnostic or treatment services for serious medical conditions,"  (LIONEL BONAVENTURE/AFP via Getty Images)

Of the 39 questions posed to ChatGPT, only 10 responses were deemed "satisfactory," according to the research team's criteria.

The study findings were presented at ASHP’s Midyear Clinical Meeting from Dec. 3 to Dec. 7 in Anaheim, California.

Grossman, the lead author, shared her initial reaction to the study's findings with Fox News Digital.

BREAST CANCER BREAKTHROUGH: AI PREDICTS A THIRD OF CASES PRIOR TO DIAGNOSIS IN MAMMOGRAPHY STUDY

Since "we had not used ChatGPT previously, we were surprised by ChatGPT’s ability to provide quite a bit of background information about the medication and/or disease state relevant to the question within a matter of seconds," she said via email. 

"Despite that, ChatGPT did not generate accurate and/or complete responses that directly addressed most questions."

Grossman also mentioned her surprise that ChatGPT was able to generate "fabricated references to support the information provided."

Out of 39 questions posed to ChatGPT, only 10 of the responses were deemed "satisfactory" according to the research team's criteria. (Frank Rumpenhorst/picture alliance via Getty Images; iStock)

In one example she cited from the study, ChatGPT was asked if "a drug interaction exists between Paxlovid, an antiviral medication used as a treatment for COVID-19, and verapamil, a medication used to lower blood pressure."

HEAD OF GOOGLE BARD BELIEVES AI CAN HELP IMPROVE COMMUNICATION AND COMPASSION: ‘REALLY REMARKABLE’

The AI model responded that no interactions had been reported with this combination.

But in reality, Grossman said, the two drugs pose a potential threat of "excessive lowering of blood pressure" when combined.

"Without knowledge of this interaction, a patient may suffer from an unwanted and preventable side effect," she warned.

"It is always important to consult with health care professionals before using information that is generated by computers."

ChatGPT should not be considered an "authoritative source of medication-related information," Grossman emphasized.

"Anyone who uses ChatGPT should make sure to verify information obtained from trusted sources — namely pharmacists, physicians or other health care providers," Grossman added.

MILITARY MENTAL HEALTH IN FOCUS AS AI TRAINING SIMULATES REAL CONVERSATIONS TO HELP PREVENT VETERAN SUICIDE

The LIU study did not evaluate the responses of other generative AI platforms, Grossman pointed out — so there isn’t any data on how other AI models would perform under the same condition.

"Regardless, it is always important to consult with health care professionals before using information that is generated by computers, which are not familiar with a patient’s specific needs," she said.

Fox News Digital reached out to OpenAI, the developer of ChatGPT, for comment on the new study.

OpenAI has a usage policy that disallows use for medical instruction, a company spokesperson previously told Fox News Digital in a statement.

Paxlovid, Pfizer's antiviral medication to treat COVID-19, is displayed in this picture illustration taken on Oct. 7, 2022. When ChatGPT was asked if a drug interaction exists between Paxlovid and verapamil, the chatbot answered incorrectly, a new study reported. (REUTERS/Wolfgang Rattay/Illustration)

"OpenAI’s models are not fine-tuned to provide medical information. You should never use our models to provide diagnostic or treatment services for serious medical conditions," the company spokesperson stated earlier this year. 

"OpenAI’s platforms should not be used to triage or manage life-threatening issues that need immediate attention."

Health care providers "must provide a disclaimer to users informing them that AI is being used and of its potential limitations." 

The company also requires that when using ChatGPT to interface with patients, health care providers "must provide a disclaimer to users informing them that AI is being used and of its potential limitations." 

In addition, as Fox News Digital previously noted, one big caveat is that ChatGPT’s source of data is the internet — and there is plenty of misinformation on the web, as most people are aware. 

That’s why the chatbot’s responses, however convincing they may sound, should always be vetted by a doctor.

The new study's author suggested consulting with a health care professional before relying on generative AI for medical inquiries. (iStock)

Additionally, ChatGPT was only "trained" on data up to September 2021, according to multiple sources. While it can increase its knowledge over time, it has limitations in terms of serving up more recent information.

Last month, CEO Sam Altman reportedly announced that OpenAI's ChatGPT had gotten an upgrade — and would soon be trained on data up to April 2023.

Dr. Harvey Castro, a Dallas, Texas-based board-certified emergency medicine physician and national speaker on AI in health care, weighed in on the "innovative potential" that ChatGPT offers in the medical arena.

"For general inquiries, ChatGPT can provide quick, accessible information, potentially reducing the workload on health care professionals," he told Fox News Digital.

ARTIFICIAL INTELLIGENCE HELPS DOCTORS PREDICT PATIENTS’ RISK OF DYING, STUDY FINDS: ‘SENSE OF URGENCY’

"ChatGPT's machine learning algorithms allow it to improve over time, especially with proper reinforcement learning mechanisms," he also said.

ChatGPT’s recently reported response inaccuracies, however, pose a "critical issue" with the program, the AI expert pointed out.

"This is particularly concerning in high-stakes fields like medicine," Castro said.

A health tech expert noted that medical professionals are responsible for "guiding and critiquing" artificial intelligence models as they evolve.  (iStock)

Another potential risk is that ChatGPT has been shown to "hallucinate" information — meaning it might generate plausible but false or unverified content, Castro warned. 

CLICK HERE TO SIGN UP FOR OUR HEALTH NEWSLETTER

"This is dangerous in medical settings where accuracy is paramount," said Castro.

"While ChatGPT shows promise in health care, its current limitations … underscore the need for cautious implementation."

AI "currently lacks the deep, nuanced understanding of medical contexts" possessed by human health care professionals, Castro added.

"While ChatGPT shows promise in health care, its current limitations, particularly in handling drug-related queries, underscore the need for cautious implementation."

OpenAI, the developer of ChatGPT, has a usage policy that disallows use for medical instruction, a company spokesperson told Fox News Digital earlier this year. (Jaap Arriens/NurPhoto via Getty Images)

Speaking as an ER physician and AI health care consultant, Castro emphasized the "invaluable" role that medical professionals have in "guiding and critiquing this evolving technology."

CLICK HERE TO GET THE FOX NEWS APP

"Human oversight remains indispensable, ensuring that AI tools like ChatGPT are used as supplements rather than replacements for professional medical judgment," Castro added.

Melissa Rudy of Fox News Digital contributed reporting. 

For more Health articles, visit www.foxnews.com/health.

Angelica Stabile is a lifestyle writer for Fox News Digital.

Why the real estate market continues to explode

Watch CBS News

As the pandemic shifted our work lives into working-from-home lives, it also accelerated trends among homebuyers toward more affordable locales. As a result, competition for houses in smaller cities, like Boise, Idaho, has reached never-before-seen heights. Correspondent David Pogue talks with realtors across the country about their unprecedented sales, and with new homeowners whose migrations from larger metropolitan areas helped fuel a residential bull market.

Be the first to know

Get browser notifications for breaking news, live events, and exclusive reporting.

ChatGPT struggles to answer medical questions, new research finds

CNN  — 

ChatGPT might not be a cure-all for answers to medical questions, a new study suggests.

Researchers at Long Island University posed 39 medication-related queries to the free version of the artificial intelligence chatbot, all of which were real questions from the university’s College of Pharmacy drug information service. The software’s answers were then compared with responses written and reviewed by trained pharmacists.

The study found that ChatGPT provided accurate responses to only about 10 of the questions, or about a quarter of the total. For the other 29 prompts, the answers were incomplete or inaccurate, or they did not address the questions.

The findings were presented Tuesday at the annual meeting of the American Society for Health-Systems Pharmacists in Anaheim, California.

ChatGPT, OpenAI’s experimental AI chatbot, was released in November 2022 and became the fastest-growing consumer application in history, with nearly 100 million people registering within two months.

Given that popularity, the researchers’ interest was sparked by concern that their students, other pharmacists and ordinary consumers would turn to resources like ChatGPT to explore questions about their health and medication plans, said Sara Grossman, an associate professor of pharmacy practice at Long Island University and one of the study’s authors.

Those queries, they found, often yielded inaccurate – or even dangerous – responses.

In one question, for example, researchers asked ChatGPT whether the Covid-19 antiviral medication Paxlovid and the blood-pressure lowering medication verapamil would react with each other in the body. ChatGPT responded that taking the two medications together would yield no adverse effects.

In reality, people who take both medications might have a large drop in blood pressure, which can cause dizziness and fainting. For patients taking both, clinicians often create patient-specific plans, including lowering the dose of verapamil or cautioning the person to get up slowly from a sitting position, Grossman said.

ChatGPT’s guidance, she added, would have put people in harm’s way.

“Using ChatGPT to address this question would put a patient at risk for an unwanted and preventable drug interaction,” Grossman wrote in an email to CNN.

When the researchers asked the chatbot for scientific references to support each of its responses, they found that the software could provide them for only eight of the questions they asked. And in each case, they were surprised to find that ChatGPT was fabricating references.

At first glance, the citations looked legitimate: They were often formatted appropriately, provided URLs and were listed under legitimate scientific journals. But when the team attempted to find the referenced articles, they realized that ChatGPT had given them fictional citations.

In one case, the researchers asked ChatGPT how to convert spinal injection doses of the muscle spasm medication baclofen to corresponding oral doses. Grossman’s team could not find a scientifically established dose conversion ratio, but ChatGPT put forth a single conversion rate and cited two medical organizations’ guidance, she said.

However, neither organization provides any official guidance on the dose conversion rate. In fact, the conversion factor that ChatGPT suggested had never been scientifically established. The software also provided an example calculation for the dose conversion but with a critical mistake: It mixed up units when calculating the oral dose, throwing off the dose recommendation by a factor of 1,000.

If that guidance was followed by a health care professional, Grossman said, they might give a patient an oral baclofen dose 1,000 times lower than required, which could cause withdrawal symptoms like hallucinations and seizures.

“There were numerous errors and “problems’ with this response and ultimately, it could have a profound impact on patient care,” she wrote.

The Long Island University study is not the first to raise concerns about ChatGPT’s fictional citations. Previous research has also documented that, when asked medical questions, ChatGPT can create deceptive forgeries of scientific references, even listing the names of real authors with previous publications in scientific journals.

Grossman, who had worked little with the software before the study, was surprised by how confidently ChatGPT was able to synthesize information nearly instantaneously, answers that would take trained professionals hours to compile.

“The responses were phrased in a very professional and sophisticated manner, and it just seemed it can contribute to a sense of confidence in the accuracy of the tool,” she said. “A user, a consumer, or others that may not be able to discern can be swayed by the appearance of authority.”

A spokesperson for OpenAI, the organization that develops ChatGPT, said it advises users not to rely on responses as a substitute for professional medical advice or treatment.

The spokesperson pointed to ChatGPT’s usage policies, which indicate that “OpenAI’s models are not fine-tuned to provide medical information.” The policy also states that the models should never be used to provide “diagnostic or treatment services for serious medical conditions.”

Although Grossman was unsure of how many people use ChatGPT to address medication questions, she raised concerns that they could use the chatbot like they would search for medical advice on search engines like Google.

“People are always looking for instantaneous responses when they have this at their fingertips,” Grossman said. “I think that this is just another approach of using ‘Dr. Google’ and other seemingly easy methods of obtaining information.”

For online medical information, she recommended that consumers use governmental websites that provide reputable information, like the National Institutes of Health’s MedlinePlus page.

Still, Grossman doesn’t believe that online answers can replace the advice of a health care professional.

“[Websites are] maybe one starting point, but they can take their providers out of the picture when looking for information about medications that are directly applicable to them,” she said. “But it may not be applicable to the patients themselves because of their personal case, and every patient is different. So the authority here should not be removed from the picture: the healthcare professional, the prescriber, the patient’s physicians.”