ChatGPT found by study to spread inaccuracies when answering medication questions

ChatGPT has been found to have shared inaccurate information regarding drug usage, according to new research.

In a study led by Long Island University (LIU) in Brooklyn, New York, nearly 75% of drug-related, pharmacist-reviewed responses from the generative AI chatbot were found to be incomplete or wrong.

In some cases, ChatGPT, which was developed by OpenAI in San Francisco and released in late 2022, provided "inaccurate responses that could endanger patients," the American Society of Health System Pharmacists (ASHP), headquartered in Bethesda, Maryland, stated in a press release.

WHAT IS ARTIFICIAL INTELLIGENCE?

ChatGPT also generated "fake citations" when asked to cite references to support some responses, the same study also found.

Along with her team, lead study author Sara Grossman, PharmD, associate professor of pharmacy practice at LIU, asked the AI chatbot real questions that were originally posed to LIU’s College of Pharmacy drug information service between 2022 and 2023.

ChatGPT, the AI chatbot created by OpenAI, generated inaccurate responses about medications, a new study has found. The company itself previously said that "OpenAI’s models are not fine-tuned to provide medical information. You should never use our models to provide diagnostic or treatment services for serious medical conditions,"  (LIONEL BONAVENTURE/AFP via Getty Images)

Of the 39 questions posed to ChatGPT, only 10 responses were deemed "satisfactory," according to the research team's criteria.

The study findings were presented at ASHP’s Midyear Clinical Meeting from Dec. 3 to Dec. 7 in Anaheim, California.

Grossman, the lead author, shared her initial reaction to the study's findings with Fox News Digital.

BREAST CANCER BREAKTHROUGH: AI PREDICTS A THIRD OF CASES PRIOR TO DIAGNOSIS IN MAMMOGRAPHY STUDY

Since "we had not used ChatGPT previously, we were surprised by ChatGPT’s ability to provide quite a bit of background information about the medication and/or disease state relevant to the question within a matter of seconds," she said via email. 

"Despite that, ChatGPT did not generate accurate and/or complete responses that directly addressed most questions."

Grossman also mentioned her surprise that ChatGPT was able to generate "fabricated references to support the information provided."

Out of 39 questions posed to ChatGPT, only 10 of the responses were deemed "satisfactory" according to the research team's criteria. (Frank Rumpenhorst/picture alliance via Getty Images; iStock)

In one example she cited from the study, ChatGPT was asked if "a drug interaction exists between Paxlovid, an antiviral medication used as a treatment for COVID-19, and verapamil, a medication used to lower blood pressure."

HEAD OF GOOGLE BARD BELIEVES AI CAN HELP IMPROVE COMMUNICATION AND COMPASSION: ‘REALLY REMARKABLE’

The AI model responded that no interactions had been reported with this combination.

But in reality, Grossman said, the two drugs pose a potential threat of "excessive lowering of blood pressure" when combined.

"Without knowledge of this interaction, a patient may suffer from an unwanted and preventable side effect," she warned.

"It is always important to consult with health care professionals before using information that is generated by computers."

ChatGPT should not be considered an "authoritative source of medication-related information," Grossman emphasized.

"Anyone who uses ChatGPT should make sure to verify information obtained from trusted sources — namely pharmacists, physicians or other health care providers," Grossman added.

MILITARY MENTAL HEALTH IN FOCUS AS AI TRAINING SIMULATES REAL CONVERSATIONS TO HELP PREVENT VETERAN SUICIDE

The LIU study did not evaluate the responses of other generative AI platforms, Grossman pointed out — so there isn’t any data on how other AI models would perform under the same condition.

"Regardless, it is always important to consult with health care professionals before using information that is generated by computers, which are not familiar with a patient’s specific needs," she said.

Fox News Digital reached out to OpenAI, the developer of ChatGPT, for comment on the new study.

OpenAI has a usage policy that disallows use for medical instruction, a company spokesperson previously told Fox News Digital in a statement.

Paxlovid, Pfizer's antiviral medication to treat COVID-19, is displayed in this picture illustration taken on Oct. 7, 2022. When ChatGPT was asked if a drug interaction exists between Paxlovid and verapamil, the chatbot answered incorrectly, a new study reported. (REUTERS/Wolfgang Rattay/Illustration)

"OpenAI’s models are not fine-tuned to provide medical information. You should never use our models to provide diagnostic or treatment services for serious medical conditions," the company spokesperson stated earlier this year. 

"OpenAI’s platforms should not be used to triage or manage life-threatening issues that need immediate attention."

Health care providers "must provide a disclaimer to users informing them that AI is being used and of its potential limitations." 

The company also requires that when using ChatGPT to interface with patients, health care providers "must provide a disclaimer to users informing them that AI is being used and of its potential limitations." 

In addition, as Fox News Digital previously noted, one big caveat is that ChatGPT’s source of data is the internet — and there is plenty of misinformation on the web, as most people are aware. 

That’s why the chatbot’s responses, however convincing they may sound, should always be vetted by a doctor.

The new study's author suggested consulting with a health care professional before relying on generative AI for medical inquiries. (iStock)

Additionally, ChatGPT was only "trained" on data up to September 2021, according to multiple sources. While it can increase its knowledge over time, it has limitations in terms of serving up more recent information.

Last month, CEO Sam Altman reportedly announced that OpenAI's ChatGPT had gotten an upgrade — and would soon be trained on data up to April 2023.

Dr. Harvey Castro, a Dallas, Texas-based board-certified emergency medicine physician and national speaker on AI in health care, weighed in on the "innovative potential" that ChatGPT offers in the medical arena.

"For general inquiries, ChatGPT can provide quick, accessible information, potentially reducing the workload on health care professionals," he told Fox News Digital.

ARTIFICIAL INTELLIGENCE HELPS DOCTORS PREDICT PATIENTS’ RISK OF DYING, STUDY FINDS: ‘SENSE OF URGENCY’

"ChatGPT's machine learning algorithms allow it to improve over time, especially with proper reinforcement learning mechanisms," he also said.

ChatGPT’s recently reported response inaccuracies, however, pose a "critical issue" with the program, the AI expert pointed out.

"This is particularly concerning in high-stakes fields like medicine," Castro said.

A health tech expert noted that medical professionals are responsible for "guiding and critiquing" artificial intelligence models as they evolve.  (iStock)

Another potential risk is that ChatGPT has been shown to "hallucinate" information — meaning it might generate plausible but false or unverified content, Castro warned. 

CLICK HERE TO SIGN UP FOR OUR HEALTH NEWSLETTER

"This is dangerous in medical settings where accuracy is paramount," said Castro.

"While ChatGPT shows promise in health care, its current limitations … underscore the need for cautious implementation."

AI "currently lacks the deep, nuanced understanding of medical contexts" possessed by human health care professionals, Castro added.

"While ChatGPT shows promise in health care, its current limitations, particularly in handling drug-related queries, underscore the need for cautious implementation."

OpenAI, the developer of ChatGPT, has a usage policy that disallows use for medical instruction, a company spokesperson told Fox News Digital earlier this year. (Jaap Arriens/NurPhoto via Getty Images)

Speaking as an ER physician and AI health care consultant, Castro emphasized the "invaluable" role that medical professionals have in "guiding and critiquing this evolving technology."

CLICK HERE TO GET THE FOX NEWS APP

"Human oversight remains indispensable, ensuring that AI tools like ChatGPT are used as supplements rather than replacements for professional medical judgment," Castro added.

Melissa Rudy of Fox News Digital contributed reporting. 

For more Health articles, visit www.foxnews.com/health.

Angelica Stabile is a lifestyle writer for Fox News Digital.

Four Questions To Define Real Success At Work

Stop. Breathe. Look at yourself and ask: “Is this what success looks like?”

Even if the answer to that question makes you feel like you’ve stopped short of the finish line, there’s hope. I was given a glimpse at what makes success and even how to measure it during my interview with Emma Avignon. CEO of Mentore Consulting, Avignon is a woman who chose to go the extra mile towards creating success – and who found her purpose in teaching others that they can, too.

Emma Avignon, CEO, Mentore Consulting LLP

Avignon has undergone a professional transformation that’s taken her from her accountant position at EY to a whole new world of accountability. She learned how to establish her value while exploring different corporate roles, and she’s cultivated a series of questions that she shared with me that has allowed her to ensure measurable, “real” success.

 1. What does success look like?

Success - it can be defined as having accomplished an end goal. While the end goal may be unique to each of us, the way we measure success is the same, and begins with the question, “What is my purpose?” This type of self-awareness is required to recognize real success.  

Avignon found her purpose by embracing change. She first worked as an analyst at PWC and then transitioned into Corporate Restructuring and Corporate Finance at EY. Positions like this gave Avignon the opportunity to deal with clients in a face-to-face way—allowing her to gauge their reactions, and to demonstrate the value she could bring as she directly impacted the organizations. She was enjoying a career full of very visible success.

“When you’re client-facing in a professional services firm, you are bringing in revenue to the firm so your value is easily recognized straight away,” says Avignon. “But when you’re not client-facing, it’s much harder.”

When she decided that the time was right to start her family, Avignon was faced with a decision.

Her current position put her on a schedule that wasn’t conducive to caring for children. The jet-setting, front facing, in demand kind of job wasn’t a good fit for her personal life, so Avignon made a career change that demonstrated the best kind of professional evolution – one that aligned with her core values.

2. What is your vision and the big picture?

True success is found in living the life that you want to live. Avignon took time to figure out the pieces needed to create her vision of success, and worked towards bringing those parts together.

Avignon moved into an internal position at EY in business development and realized two things: She needed to find another way to demonstrate the value she could bring to the firm, and she needed to do it in a way that spoke to who she was at a professional and personal level.

Avignon knew her big picture needed to include a flexible schedule, and that she had to have the agility to change the way she measured success and the way her value was showcased. She was so driven that she changed the direction of her professional purpose in a position where many might have just faded into the background.

“I didn’t want to be one of those people that others ask, “What does she actually do?” That didn’t fit with my sense of self:  I’m ambitious, I’m driven. I might not be client-facing at this point in time, but I still want to be known for adding value .”

There are times when contributions in the workplace will get overlooked. They may be more abstract or a smaller portion of the whole. This doesn’t make them any less meaningful than those that provide a more measurable worth. Know your value, Avignon says.

3. What is the outcome we are looking for?

Most of us have a love-hate relationship with maternity leave -- we love the reason but hate the professional risk. Many fear for their jobs when they have to leave for a significant amount of time. It’s been a huge decision point for women, and now many men who have the same choice to make with shared parental leave. It is a key step for many people to reevaluate our personal and professional purpose.

Avignon had discovered her definition of success and made it a visible measure of her worth within her firm. She was high achieving and didn’t let pregnancy put her off from developing her business case for the next promotion – which she got immediately prior to her second maternity leave.

“One of the senior partners said to me a piece of advice that stuck with me forever which was, “Emma, don’t measure your success on how far you get. Measure it on how far the people that you helped have gone to.”

Shortly thereafter, Avignon was introduced to Mentore Consulting, an organization with a goal to accelerate top talent within companies and to help companies achieve more diverse leadership teams. With solutions fit for both genders, Mentore Consulting has a really important component that is driven by the need to advance women within organizations. It’s about helping accelerate the progression of women and helping them then achieve the development goals that they each need for forward progression, enabling them to access the same opportunities as men.

“I’d always done mentoring and career development work within the firm because these opportunities are so readily available within these big accounting firms. I also did some pro-bono work and I ended up doing mentoring work to the EY people who were going on secondment to The Social Business Trust.”

This experience would change Avignon’s path. She began working with the woman who would introduce her to Mentore Consulting. Over the course of six months, the women corresponded and worked together periodically, and Avignon was finally offered a position at the company.   

Once again, Avignon had a new understanding of what her own “real” success looked like. “I had a view on Mentore, on what it was doing and thinking was the coolest business on the planet at that point. And I thought, “Wow!” It’s helping people achieve their potential. It’s got a focus on women, as well as focus on diversity. It has got a clear big purpose attached to it.”

4. What does great look like to you?

Having a clear view of the best possible outcome in any situation is key to heading down the right path. Avignon is now client facing again, enjoying a balance of work and time with her children. She is getting huge satisfaction from her client work and enabling others to achieve success.

Avignon’s specialized set of questions she asks anytime she works on a new project to make sure value can be demonstrated are:

These questions have helped her to realize success and are the very definition of owning our purpose in the workplace. Success is a fluid and ever-changing idea that only you can truly define for yourself. Avignon allowed her success to be shaped by her journey towards her true purpose.

ChatGPT struggles to answer medical questions, new research finds

CNN  — 

ChatGPT might not be a cure-all for answers to medical questions, a new study suggests.

Researchers at Long Island University posed 39 medication-related queries to the free version of the artificial intelligence chatbot, all of which were real questions from the university’s College of Pharmacy drug information service. The software’s answers were then compared with responses written and reviewed by trained pharmacists.

The study found that ChatGPT provided accurate responses to only about 10 of the questions, or about a quarter of the total. For the other 29 prompts, the answers were incomplete or inaccurate, or they did not address the questions.

The findings were presented Tuesday at the annual meeting of the American Society for Health-Systems Pharmacists in Anaheim, California.

ChatGPT, OpenAI’s experimental AI chatbot, was released in November 2022 and became the fastest-growing consumer application in history, with nearly 100 million people registering within two months.

Given that popularity, the researchers’ interest was sparked by concern that their students, other pharmacists and ordinary consumers would turn to resources like ChatGPT to explore questions about their health and medication plans, said Sara Grossman, an associate professor of pharmacy practice at Long Island University and one of the study’s authors.

Those queries, they found, often yielded inaccurate – or even dangerous – responses.

In one question, for example, researchers asked ChatGPT whether the Covid-19 antiviral medication Paxlovid and the blood-pressure lowering medication verapamil would react with each other in the body. ChatGPT responded that taking the two medications together would yield no adverse effects.

In reality, people who take both medications might have a large drop in blood pressure, which can cause dizziness and fainting. For patients taking both, clinicians often create patient-specific plans, including lowering the dose of verapamil or cautioning the person to get up slowly from a sitting position, Grossman said.

ChatGPT’s guidance, she added, would have put people in harm’s way.

“Using ChatGPT to address this question would put a patient at risk for an unwanted and preventable drug interaction,” Grossman wrote in an email to CNN.

When the researchers asked the chatbot for scientific references to support each of its responses, they found that the software could provide them for only eight of the questions they asked. And in each case, they were surprised to find that ChatGPT was fabricating references.

At first glance, the citations looked legitimate: They were often formatted appropriately, provided URLs and were listed under legitimate scientific journals. But when the team attempted to find the referenced articles, they realized that ChatGPT had given them fictional citations.

In one case, the researchers asked ChatGPT how to convert spinal injection doses of the muscle spasm medication baclofen to corresponding oral doses. Grossman’s team could not find a scientifically established dose conversion ratio, but ChatGPT put forth a single conversion rate and cited two medical organizations’ guidance, she said.

However, neither organization provides any official guidance on the dose conversion rate. In fact, the conversion factor that ChatGPT suggested had never been scientifically established. The software also provided an example calculation for the dose conversion but with a critical mistake: It mixed up units when calculating the oral dose, throwing off the dose recommendation by a factor of 1,000.

If that guidance was followed by a health care professional, Grossman said, they might give a patient an oral baclofen dose 1,000 times lower than required, which could cause withdrawal symptoms like hallucinations and seizures.

“There were numerous errors and “problems’ with this response and ultimately, it could have a profound impact on patient care,” she wrote.

The Long Island University study is not the first to raise concerns about ChatGPT’s fictional citations. Previous research has also documented that, when asked medical questions, ChatGPT can create deceptive forgeries of scientific references, even listing the names of real authors with previous publications in scientific journals.

Grossman, who had worked little with the software before the study, was surprised by how confidently ChatGPT was able to synthesize information nearly instantaneously, answers that would take trained professionals hours to compile.

“The responses were phrased in a very professional and sophisticated manner, and it just seemed it can contribute to a sense of confidence in the accuracy of the tool,” she said. “A user, a consumer, or others that may not be able to discern can be swayed by the appearance of authority.”

A spokesperson for OpenAI, the organization that develops ChatGPT, said it advises users not to rely on responses as a substitute for professional medical advice or treatment.

The spokesperson pointed to ChatGPT’s usage policies, which indicate that “OpenAI’s models are not fine-tuned to provide medical information.” The policy also states that the models should never be used to provide “diagnostic or treatment services for serious medical conditions.”

Although Grossman was unsure of how many people use ChatGPT to address medication questions, she raised concerns that they could use the chatbot like they would search for medical advice on search engines like Google.

“People are always looking for instantaneous responses when they have this at their fingertips,” Grossman said. “I think that this is just another approach of using ‘Dr. Google’ and other seemingly easy methods of obtaining information.”

For online medical information, she recommended that consumers use governmental websites that provide reputable information, like the National Institutes of Health’s MedlinePlus page.

Still, Grossman doesn’t believe that online answers can replace the advice of a health care professional.

“[Websites are] maybe one starting point, but they can take their providers out of the picture when looking for information about medications that are directly applicable to them,” she said. “But it may not be applicable to the patients themselves because of their personal case, and every patient is different. So the authority here should not be removed from the picture: the healthcare professional, the prescriber, the patient’s physicians.”