Loading...
Performance of Google's Artificial Intelligence Chatbot "Bard" (Now "Gemini") on Ophthalmology Board Exam Practice Questions
Citations
Altmetric:
Soloist
Composer
Publisher
Springer Science and Business Media LLC
Date
3/31/2024
Additional date(s)
Abstract
Contents
Subject
Subject(s)
Research Projects
Organizational Units
Journal Issue
Genre
Description
Purpose: To assess the performance of "Bard," one of ChatGPT's competitors, in answering practicequestions for the ophthalmology board certification exam.Methods: In December 2023, 250 multiple-choice questions from the "BoardVitals" ophthalmology examquestion bank were randomly selected and entered into Bard to assess the artificial intelligence chatbot'sability to comprehend, process, and answer complex scientific and clinical ophthalmic questions. A randommix of text-only and image-and-text questions were selected from 10 subsections. Each subsection included25 questions. The percentage of correct responses was calculated per section, and an overall assessmentscore was determined.Results: On average, Bard answered 62.4% (156/250) of questions correctly. The worst performance was 24%(6/25) on the topic of "Retina and Vitreous," and the best performance was on "Oculoplastics," with a scoreof 84% (21/25). While the majority of questions were entered with minimal difficulty, not all questions couldbe processed by Bard. This was particularly an issue for questions that included human images and multiplevisual files. Some vignette-style questions were also not understood by Bard and were therefore omitted.Future investigations will focus on having more questions per subsection to increase available data points.Conclusions: While Bard answered the majority of questions correctly and is capable of analyzing vastamounts of medical data, it ultimately lacks the holistic understanding and experience-informed knowledgeof an ophthalmologist. An ophthalmologist's ability to synthesize diverse pieces of information and drawfrom clinical experience to answer complex standardized board questions is at present irreplaceable, andartificial intelligence, in its current form, can be employed as a valuable tool for supplementing clinicians'study methods.
Format
Department
Burnett School of Medicine