ChatGPT Flubbed Drug Information Questions

ANAHEIM, Calif. -- ChatGPT provided incorrect or incomplete information when asked about drugs, and in some cases invented references to support its answers, two evaluative studies found.

In the first, 39 questions sent to a drug information service for pharmacists were later posed to ChatGPT, which provided no response, an inaccurate response, or an incomplete response to 74% of them, Tina Zerilli,, PharmD, of Long Island University in Brooklyn, New York, and colleagues reported.

For example, in a response to a real query of whether there was a drug interaction between nirmatrelvir/ritonavir (Paxlovid) and verapamil (Verelan), a blood pressure lowering drug, ChatGPT indicated that there were no interactions, though ritonavir can interact with verapamil.

Moreover, in instances where the artificial intelligence (AI) chatbot did provide a response with references, it did so each time citing references that were fabricated, with URLs that led to nonexistent studies, according to findings presented at the American Society of Health-System Pharmacists (ASHP) midyear meeting.

"It's an evolving technology. We should not rely on it right now as the definitive source of information," Zerilli told MedPage Today. "We need to verify all the information that is generated from it and know that it can spit out inaccurate information, bad information, fabricated results."

In the second study, ChatGPT missed at least half of established side effects for 26 of 30 FDA-approved drugs, Shunsuke Toyoda, PharmD, of Torrance Memorial Medical Center in California, and colleagues reported by checking the generative AI algorithm's performance against a pharmacological database.

"As you can see, [it was] mostly inaccurate, and as pharmacists, we always have to be 100% accurate," Toyoda told MedPage Today at a poster presentation, also at the ASHP midyear meeting. "So I'm sorry, ChatGPT -- no way they can replace us, not in our lifetime."

ChatGPT is a large language model (LLM) generative AI chatbot whose use has exploded in popularity since launching in 2022. Its role in healthcare, along with other AI tools geared toward medical professionals and industry, is still being debated even as billions of dollars in investing are poured into the space.

"We go through so many trainings, even after school, and I think, if some computer program really just came out one night and also claimed to completely replace us? I find that very insulting," said Toyoda.

Even so, the present studies offer a limited view of what AI in medicine is capable of, commented John Ayers, PhD, MA, of the Qualcomm Institute at the University of California San Diego in La Jolla, who was not involved in the study.

"They use a generic LLM that's not optimized to assess or evaluate healthcare relevant information. It's not trained specifically on that kind of data," he said. Referring to the drug information service study, Ayers said he was "surprised that it did so well, and it shows the potential, with optimization, of what could be achieved."

But at the same time, the accuracy of AI tools being used in medicine are irrelevant if regulatory bodies like the FDA don't create a framework or standards for their quality, he noted. "In a way, there's no money to do these types of evaluations," he said. "The technology companies themselves would not want to pay for this kind of research because they don't want to report if it doesn't work. And the FDA is not going to want to mandate this type of research."

Researchers in the drug information service study randomly assigned 39 questions they had received from January 2022 to April 2023 to one of two investigators, who created responses to the questions based on a literature search. The questions were posed to ChatGPT, followed by the phrase, "Please provide references to support the response."

Investigators evaluated the response as "satisfactory" or "unsatisfactory" based on a tool they designed, and if they disagreed, a third investigator weighed in. "Unsatisfactory" responses could include no direct response, inaccurate information, incomplete information, or extraneous information. "Satisfactory" responses were accurate and complete with no irrelevant information.

For the side effects study, researchers randomly selected 30 FDA-approved drugs and input into ChatGPT, "What are the most common side effects of [each selected drug]?" from April to June this year. ChatGPT responses that matched all common side effects listed in Lexicomp, the drug database service, were classified "accurate," those that matched half were classified "partially accurate," and those that matched less than half were "inaccurate."

Study limitations included a lack of validated tools to measure the accuracy of ChatGPT, and using versions of ChatGPT that may not be the most current.

Correction: Quotes in this story are attributable to Tina Zerilli, PharmD, not co-investigator Sara Grossman, PharmD.

Disclosures

Zerilli and Toyoda disclosed no conflicts of interest.

Primary Source

American Society of Health-System Pharmacists

Source Reference: Matsuura M, et al "Evaluation of side effect drug information generated by ChatGPT" ASHP 2023; Abstract 8-023.

Secondary Source

American Society of Health-System Pharmacists

Source Reference: Grossman S, et al "ChatGPT: evaluation of its ability to respond to drug information questions" ASHP 2023; Abstract 8-021.

Please enable JavaScript to view the

Exam4 Instructions - Day of Exam

Open and set up the Exam4 software:

Do not begin typing until the proctor announces the beginning of the exam.

Once the proctor has started the exam, you may turn past the first page and begin the exam. Type your exam answers in the Exam4 software. If your exam has multiple-choice questions, you answer them in the Exam4 software by toggling between the essay window and the multiple-choice window. 

Finish and Submit your Exam (You will NOT be able to make changes to your exam after this step)

If your laptop encounters a problem during the exam, notify the proctor immediately. All proctors have been trained to address commonly occurring Exam4 issues. You will have two options:

CISSP Certification Requirements And Cost

Editorial Note: We earn a commission from partner links on Forbes Advisor. Commissions do not affect our editors' opinions or evaluations.

The Certified Information Systems Security Professional (CISSP®) credential demonstrates mastery of developing and overseeing large-scale cybersecurity programs. When it comes to the best cybersecurity certifications, many consider CISSP the industry’s gold standard. Individuals who meet CISSP requirements can earn the certification and qualify to take on more professional responsibility in their cybersecurity jobs.

This guide offers information on CISSP certification requirements, including experience, suggested preparation times and CISSP certification exam costs.

CISSP certification, offered by (ISC)², is an advanced credential for information systems and cybersecurity professionals. This certification highlights an individual’s ability to create, deploy and manage cybersecurity efforts for large organizations.

CISSP certification requirements include a significant amount of professional experience and passing a lengthy exam. This credential suits experienced workers over entry-level and mid-level professionals.

Though this certification is not required by employers, it can boost candidates’ earning power and help them qualify for advanced roles in information security. CISSPs often work in positions like chief information security officer (CISO), network architect, security auditor and security manager, among others.

Aspiring CISSPs should familiarize themselves with the certification’s requirements before pursuing this credential.

CISSP certification requirements stipulate that each candidate must have a minimum of five years of relevant professional experience. (ISC)² specifies eight security domains:

Prospective CISSPs must accumulate experience in two of the eight domains to meet CISSP certification requirements. They can also apply (ISC)²-approved four-year college degrees and other credentials, which may qualify as a year of experience. Paid and unpaid internships also count toward the CISSP requirement.

The CISSP exam covers the eight domains of security in a four-hour test comprising 125 to 175 required responses. These appear as multiple-choice questions and advanced innovative items. Test-takers must earn a 700 out of 1,000 to pass. Candidates register to take the exam with Pearson VUE.

After passing the exam, individuals can apply for endorsement online. The endorsement must come from an (ISC)²-certified professional who can advocate for your professional experience as a credential-holder in good standing. Individuals must receive endorsements within nine months of passing the exam.

Earning CISSP certification can deliver many professional benefits, but individuals should also understand the costs associated with pursuing the credential. Along with the required time investment, consider the following CISSP certification exam costs and any required payments relating to preparation and recertification.

Many organizations offer prep courses for the CISSP certification exam, and their costs vary drastically. Make sure to include exam prep costs, which may range from a few hundred dollars to a few thousand, in your CISSP budget. (ISC)² offers several exam prep methods, including self-paced, instructor-led and team-based options.

The CISSP certification exam costs $749. Individuals can receive vouchers from partner organizations after completing CISSP exam training courses.

Individuals must meet CISSP recertification requirements every three years to maintain their credentials. Each certification holder must earn 120 continuing professional education (CPE) credits over this three-year period. Costs relating to CPEs can vary significantly, but each certified individual must pay an annual $125 fee to (ISC)².

Individuals who have met CISSP requirements and earned their credentials can pursue work in many information security and cybersecurity roles. As of 2023, the number of open cybersecurity roles far outpaces the number of qualified professionals, indicating strong continued demand in the sector.

We sourced salary information for this section from Payscale.

Data from Cyberseek.org indicates that among current cybersecurity openings requiring certification, CISSP ranks as the most in-demand credential. The following section explores roles for CISSP certification-holders.

Average Annual Salary: Around $173,500Required Education: Bachelor’s degree in cybersecurity, information security or a related field; master’s preferredJob Description: CISOs rank among the top positions in information security for responsibility and salary. This C-suite role reports directly to the CEO and requires significant experience, practical skills and expertise in information security.

CISOs oversee their organizations’ information security efforts. Often referred to as “chief security officers,” they supervise teams of infosec workers, set organizational directives, establish company-wide best practices and manage resource allotment. CISOs working in large, international businesses may interact with government agencies and congresspeople to ensure compliance with legal standards for information security.

Average Annual Salary: Around $125,000Required Education: Bachelor’s degree in computer science or related field, MBA degree often preferredJob Description: IT directors oversee departments of IT workers and manage organizations’ computer systems operations. They provide solutions to companies’ computer-related issues, including software upgrades, security concerns and general technical issues. IT directors communicate with executives to ensure company-wide directives are carried out successfully.

These directors research new IT software and hardware to keep their organizations up to date and safe. They track metrics for managing IT professionals, along with storage, hardware and software. IT directors also handle employee schedules and budget planning. As department heads, they must possess strong communication skills to interact with mid-level professionals and C-suite executives.

Average Annual Salary: Around $73,500Required Education: Bachelor’s degree in cybersecurity, computer and information technology or a related fieldJob Description: Security analysts work in computer systems, networks and information security departments to prevent, monitor and respond to security breaches. This broad professional title refers to workers who handle a variety of tasks in computer and network security.

These professionals work in many industries as “first responders” for cyberattacks. They must demonstrate deep knowledge of hardware, software and data storage to understand potential vulnerabilities and security solutions. Security analysts may help design security systems and handle encryption efforts for businesses to protect sensitive information.

Average Annual Salary: Around $126,000Required Education: Bachelor’s degree in network engineering or a related field; master’s in cybersecurity or a related field often preferredJob Description: Network architects design and implement organizations’ security infrastructures. These professionals test and analyze existing safety structures to identify vulnerabilities and improvements. They install and maintain computer systems, including interconnected devices like firewalls and routers.

Before deploying any updates or upgrades, these information security professionals create models to test their networks in a controlled environment. Modeling allows network architects to forecast security and traffic issues before implementing their infrastructures in the real world. They also train and educate IT workers on organizational best practices.

The two primary CISSP requirements are passing the exam and gaining five years of relevant professional experience.

No. CISSP certification suits experienced cybersecurity and information security professionals, requiring a minimum of five years of experience in the field.