What Insurers Look for in a Life Insurance Blood Test & Exam

Study finds AI language model failed to produce appropriate questions, answers for medical school exam

With concerns mounting that artificial intelligence (AI) could have a profound impact on traditional teaching in academic settings, many question the role of ChatGPT, a sophisticated AI language model that can generate content that mimics human conversation.

ChatGPT offers the potential to assist or take over the student writing process with the capability of authoring everything from college admissions essays to term papers. But, can it also be used to aid the prodigious, sometimes daunting learning process in the medical school curriculum?

Researchers from Boston University Chobanian & Avedisian School of Medicine used ChatGPT to create multiple-choice questions, along with explanations of correct and incorrect choices, for a graduate and medical school immunology class that was taught by faculty in the school's department of pathology & laboratory medicine. They found the AI language model wrote acceptable questions but failed to produce appropriate answers.

The study is published in the journal Academic Pathology.

"Unfortunately, ChatGPT only generated correct questions and answers with explanations in 32% of the questions (19 out of 60 individual questions). In many instances, ChatGPT failed to provide an explanation for the incorrect answers. An additional 25% of the questions had answers that were either wrong or misleading," explained corresponding author Daniel Remick, MD, professor of pathology & laboratory medicine at the school

According to the researchers, students appreciate practice exams that can be used to study for their actual exams. These practice exams have even greater utility when explanations for answers are included since students will learn the rationale for the correct answer and have explanations for the incorrect answers.

Since ChatGPT generated questions with vague or confusing question stems and poor explanations of the answer choices, this study tool may not be entirely viable.

"These types of misleading questions may create further confusion about the topics, especially since the students have not gained expertise and they may not be able to find errors in the questions. "However, despite the issues we encountered, instructors may still find ChatGPT useful for creating practice exams with explanations—with the caveat that extensive editing may be required," added Remick.

More information: Alexander Ngo et al, ChatGPT 3.5 fails to write appropriate multiple choice practice exam questions, Academic Pathology (2023). DOI: 10.1016/j.acpath.2023.100099

Citation: Study finds AI language model failed to produce appropriate questions, answers for medical school exam (2023, December 20) retrieved 6 January 2024 from https://medicalxpress.com/news/2023-12-ai-language-medical-school-exam.html

This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

New Cardiology Certification Board: What's the Plan?

The proposal by the major cardiovascular societies in the US to form a new board of cardiovascular medicine to manage initial and ongoing certification of cardiologists represents something of a revolution in the field of continuing medical education and exam of competency. 

Five US cardiovascular societies — the American College of Cardiology (ACC), the American Heart Association (AHA), the Heart Failure Society of America (HFSA), the Heart Rhythm Society (HRS), and the Society for Cardiovascular Angiography & Interventions (SCAI) — have now joined forces to propose a new professional certification board for cardiovascular medicine, to be known as the American Board of Cardiovascular Medicine (ABCVM). 

The ABCVM would be independent of the American Board of Internal Medicine (ABIM), the current organization providing maintenance of certification for cardiologists as well as many other internal medicine subspecialties. The ABIM's maintenance of certification process has been widely criticized for many years and has been described as "needlessly burdensome and expensive." 

The ABCVM is hoping to offer a more appropriate and supportive approach, according to Jeffrey Kuvin, MD, a trustee of the ACC, who has been heading up the working group to develop this plan. 

Kuvin, who is chair of the cardiology at Northwell Health, Manhasset, New York, a l arge academic healthcare system, explained that maintenance of certification has been a topic of discussion across the cardiovascular community for many years, and the ACC has a working group focused on the next steps for evaluation of competency, which he chairs.

"The topic of evaluation of competence has been on the mind of the ACC for many years and hence a work group was developed to focus on this," Kuvin noted. "A lot of evolution of the concepts and next steps have been drawn out of this working group. And now other cardiovascular societies have joined to show unification across the house of cardiology and that this is indeed the way that the cardiovascular profession should move." 

The general concept behind the new cardiology board is to separate cardiology from the ABIM. 

"This is rooted from the concept that cardiology has evolved so much over the last few decades into such a large multidimensional specialty that it really does demarcate itself from internal medicine, and as such, it deserves a separate board governed by cardiologists with collaboration across the entirely of cardiology," Kuvin said. 

Cardiology has had significant growth and expansion of technology, tools, medications, and the approach to patients in many specialities and subspecialties, he added. "We have defined training programs in many different areas within cardiology; we have our own guidelines, our own competency statements, and in many cases, cardiology exists as its own department outside of medicine in many institutions. It's just time to separate cardiology from the umbrella of internal medicine." 

The new cardiology board would be separate from, and not report to, the ABIM; rather, it would report directly to the American Board of Medical Specialties (ABMS), the only recognized medical certification body in the US. 

Under the present system, managed by the ABIM, clinicians must undergo two stages of certification to be a cardiologist. First, they have to pass the initial certification exam in general cardiology, and then exams in one of four subspecialties if they plan to enter one of these, including interventional cardiology, electrophysiology, advanced heart failure or adult congenital heart disease. 

Next, clinicians enter the maintenance of certification phase, which can take three different forms: 1) taking another recertification exam every 10 years; 2) the collaborative maintenance pathway — a collaboration between ACC and ABIM, which includes evaluation, learning and a certified exam each year; or 3) longitudinal knowledge and exam — in which the program interacts with the clinician on an ongoing basis, sending secured questions regularly. 

All three of these pathways for maintenance of certification involve high stakes questions and a set bar for passing or failing. 

Under the proposed new cardiology board, an initial certification exam would still be required after fellowship training, but the maintenance of certification process would be completely restructured, with the new approach taking the form of continuous learning and exam of competency. 

"This is an iterative process, but we envision with a new American Board of Cardiovascular Medicine, we will pick up where the ABIM left off," Kuvin notes. "That includes an initial certifying examination for the five areas that already exist under the ABIM system but with the opportunities to expand that to further specialties as well."

He points out that there are several areas in cardiology that are currently not represented by these five areas that warrant some discussion, including multimodality imaging, vascular heart disease, and cardio-oncology. 

"At present, everybody has to pass the general cardiology exam and then some may wish to further train and get certified in one of the other four other specific areas. But one topic that has been discussed over many years is how do we maintain competency in the areas in which clinicians practice over their lifetime as a cardiologist," Kuvin commented. 

He said the proposed cardiology board would like to adhere to some basic principles that are fundamental to the practice of medicine. 

"We want to make sure that we are practicing medicine so that our patients derive the most benefit from seeing a cardiologist," he said. "We also want to make sure, however, that this is a supportive process, supporting cardiologists to learn what they know and more importantly what they don't know; to identify knowledge gaps in specific area; to help the cardiologist fill those knowledge gaps; to acknowledge those gaps have been filled; and then move on to another area of interest. This will be the focus of this new and improved model of continuous competency."

The proposed new board also says it wants to make sure this is appropriate to the area in which the clinician is practicing.

"To take a closed book certified exam every 10 years on the world of cardiology as happens at the current time – or the exams conducted in the other two pathways – is often meaningless to the cardiologist," Kuvin says. "All three current pathways involve high stakes questions that are often irrelevant to one’s clinical practice." 

"The crux of the changes we are proposing will be away from the focus of passing a test towards a model of helping the individual with their competency, with continuous learning and evaluation of competency to help the clinician fill in their knowledge gaps," he explains.

He described the new approach as "lifelong learning," adding that, instead of it being "a punitive pass/fail environment with no feedback, which causes a lot of discontent among clinicians," it will be a supportive process, where a clinician will be helped in filling their knowledge gaps. 

"I think this would be a welcome change not just for cardiology but across medical specialties," Kuvin said. 

He also pointed out the ABMS itself is considering a continuous competency approach, and the proposed new cardiology board aims to work with the ABMS to make sure that their goals of continuous competency exam are matched. 

"The world has changed. The ability to access information has changed. It is no longer imperative for a clinician to have every piece of knowledge in their brain, but rather to know how to get knowledge and to incorporate that knowledge into clinical practice," Kuvin noted. "Competency should not involve knowledge alone as in a closed book exam. It is more about understanding the world that we live in, how to synthesize information, where we need to improve knowledge and how to do that." 

Kuvin acknowledged that asking clinicians questions is a very helpful tool to identify their knowledge base and their knowledge gaps. "But we believe the clinician needs to be given resources – that could be a conference, an article, a simulation - to fill that knowledge gap. Then we could ask clinicians some different questions and if they get those right then we have provided a service." 

Tactile skills for cardiologists needing to perform procedures – such as interventionalists or electrophysiologists may be incorporated by simulation in a technology-based scenario.

On how often these exams would take place, Kuvin said that hadn't been decided for sure. 

"We certainly do not think an exam every 10 years is appropriate. We envision, instead of an episodic model, it will be rather a lifelong journey of education and competency. This will involve frequent contact and making sure knowledge gaps are being filled. There are criteria being set out by the ABMS that there should be a certain number of touch points with individuals on an annual as well as a 5-year basis to make sure cardiologists are staying within specific guardrails. The exact nature of these is yet to be determined," he said. 

Kuvin added that it was not known yet what sort of hours would be required but added that "this will not be a significant time burden."

The application to the ABMS for a separate cardiology board is still ongoing and has not yet received formal acceptance. Representatives from the five US cardiovascular societies are in the initial stages of formulating a transition board. 

"The submission to the ABMS will take time for them to review. This could take up to a year or so," Kuvin estimates. 

This is the first time the ABMS has entertained the concept of a new board in many years, he noted. "It will be a paradigm shift for the whole country. I think that cardiology is really at the forefront and in a position where we can actually do this. If cardiovascular medicine is granted a new board, I think this will help change the approach of how physicians are assessed in terms of continuous competency not just in cardiology but across all specialties of medicine."

He added: "We are confident that we can work within the construct of the ABMS guidelines that have been revised to be much more holistic in the approach of continuous competence across the board. This includes thinking beyond rote medical knowledge and thinking about the clinician as a whole and their abilities to communicate, act professionally, work within a complex medical system, utilize medical resources effectively. These all have to be part of continuous competence."

Noting that the ABIM has received criticism over the costs of the certification process, Kuvin said they intend to make this "as lean a machine as possible with the focus on reducing the financial [burden] as well as the time burden for cardiologists. It is very important that this is not cumbersome, that it is woven into clinical practice, and that it is not costly." 

But he pointed out that building a new board will have significant costs. 

"We have to think about developing initial board certification examinations as well as changing the paradigm on continuous certification," he said. "This will take some up-front costs, and our society partners have decided that they are willing to provide some start-up funds for this. We anticipate the initial certification will remain somewhat similar in price, but the cost of ongoing continuous competency exam will be significantly reduced compared to today's models."

Kuvin said the collaboration of the five participating US cardiovascular societies was unprecedented. But he noted that while the transition board is beginning with representatives of these individual societies, it will ultimately be independent from these societies and have its own board of directors. 

He suggested that other societies representing other parts of cardiology are also interested. "Cardiology has recognized how important this is," he said. "Everybody is excited about this."