PURPOSE
Bard by Google, a direct competitor to ChatGPT, was recently released. Understanding the relative performance of these different chatbots can provide important insight into their strengths and weaknesses as well as which roles they are most suited to fill. In this project, we aimed to compare the most recent version of ChatGPT, ChatGPT-4, and Bard by Google, in their ability to accurately respond to radiology board examination practice questions.
METHODS
Text-based questions were collected from the 2017-2021 American College of Radiology's Diagnostic Radiology In-Training (DXIT) examinations. ChatGPT-4 and Bard were queried, and their comparative accuracies, response lengths, and response times were documented. Subspecialty-specific performance was analyzed as well.
RESULTS
318 questions were included in our analysis. ChatGPT answered significantly more accurately than Bard (87.11% vs 70.44%, P < .0001). ChatGPT's response length was significantly shorter than Bard's (935.28 ± 440.88 characters vs 1437.52 ± 415.91 characters, P < .0001). ChatGPT's response time was significantly longer than Bard's (26.79 ± 3.27 seconds vs 7.55 ± 1.88 seconds, P < .0001). ChatGPT performed superiorly to Bard in neuroradiology, (100.00% vs 86.21%, P = .03), general & physics (85.39% vs 68.54%, P < .001), nuclear medicine (80.00% vs 56.67%, P < .01), pediatric radiology (93.75% vs 68.75%, P = .03), and ultrasound (100.00% vs 63.64%, P < .001). In the remaining subspecialties, there were no significant differences between ChatGPT and Bard's performance.
CONCLUSION
ChatGPT displayed superior radiology knowledge compared to Bard. While both chatbots display reasonable radiology knowledge, they should be used with conscious knowledge of their limitations and fallibility. Both chatbots provided incorrect or illogical answer explanations and did not always address the educational content of the question.
[1]
Christian B. van der Pol,et al.
Using AI chatbots as a radiologic decision-making tool for liver imaging: do ChatGPT and Bard communicate information consistent with the American College of Radiology Appropriateness Criteria?
,
2023,
Journal of the American College of Radiology : JACR.
[2]
Birgit B Ertl-Wagner,et al.
Accuracy of Information and References Using ChatGPT-3 for Retrieval of Clinical Radiological Information.
,
2023,
Canadian Association of Radiologists journal = Journal l'Association canadienne des radiologistes.
[3]
R. Taylor,et al.
How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment
,
2023,
JMIR medical education.
[4]
Lu Xu,et al.
Chatbot for Health Care and Oncology Applications Using Artificial Intelligence and Machine Learning: Systematic Review
,
2021,
JMIR cancer.
[5]
D. Hommes,et al.
An Exploration Into the Use of a Chatbot for Patients With Inflammatory Bowel Diseases: Retrospective Cohort Study
,
2020,
Journal of medical Internet research.