ChatGPT found by study to spread inaccuracies when answering medication questions

ChatGPT has been found to have shared inaccurate information regarding drug usage, according to new research.

In a study led by Long Island University (LIU) in Brooklyn, New York, nearly 75% of drug-related, pharmacist-reviewed responses from the generative AI chatbot were found to be incomplete or wrong.

In some cases, ChatGPT, which was developed by OpenAI in San Francisco and released in late 2022, provided "inaccurate responses that could endanger patients," the American Society of Health System Pharmacists (ASHP), headquartered in Bethesda, Maryland, stated in a press release.

WHAT IS ARTIFICIAL INTELLIGENCE?

ChatGPT also generated "fake citations" when asked to cite references to support some responses, the same study also found.

Along with her team, lead study author Sara Grossman, PharmD, associate professor of pharmacy practice at LIU, asked the AI chatbot real questions that were originally posed to LIU’s College of Pharmacy drug information service between 2022 and 2023.

ChatGPT, the AI chatbot created by OpenAI, generated inaccurate responses about medications, a new study has found. The company itself previously said that "OpenAI’s models are not fine-tuned to provide medical information. You should never use our models to provide diagnostic or treatment services for serious medical conditions,"  (LIONEL BONAVENTURE/AFP via Getty Images)

Of the 39 questions posed to ChatGPT, only 10 responses were deemed "satisfactory," according to the research team's criteria.

The study findings were presented at ASHP’s Midyear Clinical Meeting from Dec. 3 to Dec. 7 in Anaheim, California.

Grossman, the lead author, shared her initial reaction to the study's findings with Fox News Digital.

BREAST CANCER BREAKTHROUGH: AI PREDICTS A THIRD OF CASES PRIOR TO DIAGNOSIS IN MAMMOGRAPHY STUDY

Since "we had not used ChatGPT previously, we were surprised by ChatGPT’s ability to provide quite a bit of background information about the medication and/or disease state relevant to the question within a matter of seconds," she said via email. 

"Despite that, ChatGPT did not generate accurate and/or complete responses that directly addressed most questions."

Grossman also mentioned her surprise that ChatGPT was able to generate "fabricated references to support the information provided."

Out of 39 questions posed to ChatGPT, only 10 of the responses were deemed "satisfactory" according to the research team's criteria. (Frank Rumpenhorst/picture alliance via Getty Images; iStock)

In one example she cited from the study, ChatGPT was asked if "a drug interaction exists between Paxlovid, an antiviral medication used as a treatment for COVID-19, and verapamil, a medication used to lower blood pressure."

HEAD OF GOOGLE BARD BELIEVES AI CAN HELP IMPROVE COMMUNICATION AND COMPASSION: ‘REALLY REMARKABLE’

The AI model responded that no interactions had been reported with this combination.

But in reality, Grossman said, the two drugs pose a potential threat of "excessive lowering of blood pressure" when combined.

"Without knowledge of this interaction, a patient may suffer from an unwanted and preventable side effect," she warned.

"It is always important to consult with health care professionals before using information that is generated by computers."

ChatGPT should not be considered an "authoritative source of medication-related information," Grossman emphasized.

"Anyone who uses ChatGPT should make sure to verify information obtained from trusted sources — namely pharmacists, physicians or other health care providers," Grossman added.

MILITARY MENTAL HEALTH IN FOCUS AS AI TRAINING SIMULATES REAL CONVERSATIONS TO HELP PREVENT VETERAN SUICIDE

The LIU study did not evaluate the responses of other generative AI platforms, Grossman pointed out — so there isn’t any data on how other AI models would perform under the same condition.

"Regardless, it is always important to consult with health care professionals before using information that is generated by computers, which are not familiar with a patient’s specific needs," she said.

Fox News Digital reached out to OpenAI, the developer of ChatGPT, for comment on the new study.

OpenAI has a usage policy that disallows use for medical instruction, a company spokesperson previously told Fox News Digital in a statement.

Paxlovid, Pfizer's antiviral medication to treat COVID-19, is displayed in this picture illustration taken on Oct. 7, 2022. When ChatGPT was asked if a drug interaction exists between Paxlovid and verapamil, the chatbot answered incorrectly, a new study reported. (REUTERS/Wolfgang Rattay/Illustration)

"OpenAI’s models are not fine-tuned to provide medical information. You should never use our models to provide diagnostic or treatment services for serious medical conditions," the company spokesperson stated earlier this year. 

"OpenAI’s platforms should not be used to triage or manage life-threatening issues that need immediate attention."

Health care providers "must provide a disclaimer to users informing them that AI is being used and of its potential limitations." 

The company also requires that when using ChatGPT to interface with patients, health care providers "must provide a disclaimer to users informing them that AI is being used and of its potential limitations." 

In addition, as Fox News Digital previously noted, one big caveat is that ChatGPT’s source of data is the internet — and there is plenty of misinformation on the web, as most people are aware. 

That’s why the chatbot’s responses, however convincing they may sound, should always be vetted by a doctor.

The new study's author suggested consulting with a health care professional before relying on generative AI for medical inquiries. (iStock)

Additionally, ChatGPT was only "trained" on data up to September 2021, according to multiple sources. While it can increase its knowledge over time, it has limitations in terms of serving up more recent information.

Last month, CEO Sam Altman reportedly announced that OpenAI's ChatGPT had gotten an upgrade — and would soon be trained on data up to April 2023.

Dr. Harvey Castro, a Dallas, Texas-based board-certified emergency medicine physician and national speaker on AI in health care, weighed in on the "innovative potential" that ChatGPT offers in the medical arena.

"For general inquiries, ChatGPT can provide quick, accessible information, potentially reducing the workload on health care professionals," he told Fox News Digital.

ARTIFICIAL INTELLIGENCE HELPS DOCTORS PREDICT PATIENTS’ RISK OF DYING, STUDY FINDS: ‘SENSE OF URGENCY’

"ChatGPT's machine learning algorithms allow it to improve over time, especially with proper reinforcement learning mechanisms," he also said.

ChatGPT’s recently reported response inaccuracies, however, pose a "critical issue" with the program, the AI expert pointed out.

"This is particularly concerning in high-stakes fields like medicine," Castro said.

A health tech expert noted that medical professionals are responsible for "guiding and critiquing" artificial intelligence models as they evolve.  (iStock)

Another potential risk is that ChatGPT has been shown to "hallucinate" information — meaning it might generate plausible but false or unverified content, Castro warned. 

CLICK HERE TO SIGN UP FOR OUR HEALTH NEWSLETTER

"This is dangerous in medical settings where accuracy is paramount," said Castro.

"While ChatGPT shows promise in health care, its current limitations … underscore the need for cautious implementation."

AI "currently lacks the deep, nuanced understanding of medical contexts" possessed by human health care professionals, Castro added.

"While ChatGPT shows promise in health care, its current limitations, particularly in handling drug-related queries, underscore the need for cautious implementation."

OpenAI, the developer of ChatGPT, has a usage policy that disallows use for medical instruction, a company spokesperson told Fox News Digital earlier this year. (Jaap Arriens/NurPhoto via Getty Images)

Speaking as an ER physician and AI health care consultant, Castro emphasized the "invaluable" role that medical professionals have in "guiding and critiquing this evolving technology."

CLICK HERE TO GET THE FOX NEWS APP

"Human oversight remains indispensable, ensuring that AI tools like ChatGPT are used as supplements rather than replacements for professional medical judgment," Castro added.

Melissa Rudy of Fox News Digital contributed reporting. 

For more Health articles, visit www.foxnews.com/health.

Angelica Stabile is a lifestyle writer for Fox News Digital.

Assisted living chat: Is a 10% increase per year typical?

Yeganeh Torbati, Julie Zauzmer Weil and Douglas MacMillan are reporters on The Washington Post’s financial investigations desk. Christopher Rowland is a business reporter who has recently been focused on the long-term-care industry, including costs, care shortages and evictions.

We answered your questions about assisted-living facilities on Tuesday. Here are some of the questions we answered:

The chat is over, but you can read a transcript of it below. Please note: Questions may be edited for accuracy and clarity.

Alexandra Pannoni, newsroom talent and community editor, produced this live chat.

Biden ignores reporter questions after son Hunter defies congressional subpoena

President Biden ignored reporter questions on Wednesday shortly after his son, Hunter, defied a congressional subpoena regarding the family's business dealings. 

The president's scheduled news conference before a meeting of his National Infrastructure Advisory Council was delayed for more than an hour Wednesday until Biden made an appearance, walking out to address members of the press gathered to hear the commander in chief speak from the White House.

Earlier in the day, Hunter Biden had arrived on Capitol Hill — not to comply with his subpoena and be deposed by the House Oversight Committee but instead to hold a press conference and again offer to testify publicly. He maintained that his father "was not financially involved" in his business, saying there is "no evidence because it did not happen." 

At the White House, President Biden joked with the waiting press to "please say seated," before staying on script about this administration's infrastructure initiatives. 

"Last year, I asked this council to ensure that resilience is built into all of our infrastructure projects, including critical sectors like energy, communications, transportation and health care. Together, you delivered, especially on these key challenges," Biden said. "Today, I'm looking forward to hearing about new projects this council will pursue in the coming years as well. So thank you all for being here. I'm just going to ask the press to step out so we can begin our briefing. Thank you. Thank you. Thank you." 

HUNTER BIDEN WILL NOT SIT FOR DEPOSITION BY GOP, SAYS FATHER NOT 'FINANCIALLY' INVOLVED IN HIS BUSINESS

President Biden speaks during a meeting of the National Infrastructure Advisory Council in the Indian Treaty Room on the White House campus in Washington, D.C., on Wednesday. (AP Photo/Evan Vucci)

Fox News' Peter Doocy could be heard repeatedly shouting to Biden, "Mr. President, did you watch Hunter this morning?" 

Over the sound of clamoring press, another reporter was heard asking, "Mr. President, should your son have defied the subpoena?" 

Yet, Biden looked at the press and walked away from the podium. 

Biden at the top of his brief remarks championed how two years ago he "signed into law a once-in a-generation investment in our nation's infrastructure and — to fix our roads, railroads, bridges, ports, airports, remove every single lead pipe in the country and extend high-speed internet, advance clean energy, and modernize the electric grid — energy grid."

Hunter Biden talks to reporters outside the U.S. Capitol in Washington, D.C., on Wednesday, lashing out at Republican investigators who have been digging into his business dealings.  (AP Photo/Jose Luis Magana)

JOHNSON DEFENDS VOTE TO FORMALIZE BIDEN IMPEACHMENT INQUIRY AMID WHITE HOUSE ‘IMPASSE’: 'WE HAVE NO CHOICE'

"Already, we've announced over 40,000 projects in 4,500 communities all across our country. And when folks see these big projects in their hometowns, when they see the cranes up in the air, the shovels in the ground, I really think — coming from an area that was shut down, up in Scranton, Pennsylvania, the steel mill in Claymont, Delaware — I think it brings them hope," Biden said. "Because it's not just about building an infrastructure. It's about building better infrastructure, stronger infrastructure, infrastructure to withstand 21st century challenges from climate change, cyberattacks to natural disasters to foreign threats, and so much more." 

The House is expected to vote Wednesday on a resolution to formalize the Biden impeachment inquiry, as Speaker Mike Johnson, R-La., and Republican leadership have accused the White House of "stonewalling" their investigations into the president's alleged involvement in his son Hunter's business dealings.

From left: House Judiciary Committee Chair Jim Jordan, R-Ohio; Rep. Marjorie Taylor Greene, R-Ga.; and House Oversight and Accountability Committee Chair James Comer, R-Ky., speak to reporters after Hunter Biden defied a congressional subpoena to appear privately for a deposition on Wednesday. (AP Photo/J. Scott Applewhite)

On Capitol Hill on Wednesday, Hunter Biden railed against  "MAGA" Republicans who have "invaded" his privacy, "attacked" his family and "ridiculed my struggle with addiction." 

CLICK HERE TO GET THE FOX NEWS APP

House Republicans say the Justice Department has refused to allow two attorneys to testify before the House Judiciary Committee. They claim the White House sent House Oversight and Accountability Chair Rep. James Comer, R-Ky., and House Judiciary Chair Rep. Jim Jordan, R-Ohio, a letter saying they have no intention of complying with GOP subpoenas and requests for interviews without a formal vote, and the National Archives has withheld thousands of pages of documents and emails.

Fox News' Brooke Singman contributed to this report. 

Danielle Wallace is a reporter for Fox News Digital covering politics, crime, police and more. Story tips can be sent to danielle.wallace@fox.com and on Twitter: @danimwallace.