Performance of ChatGPT and GPT-4 on Polish National Specialty Exam (NSE) in Ophthalmology

From its beginnings in the mid-20th century, artificial intelligence (AI) has experienced cycles of enthusiastic progress and quieter periods. Nevertheless, in recent years, the field has seen a revival in research and development, largely driven by the significant growth in computing capabilities and the emergence of big data. Technologies such as machine learning, neural networks, and deep learning aim to construct models capable of handling extensive datasets, recognizing patterns, and then producing insights or predictions, frequently outperforming human abilities in particular tasks [1].

The swift progression of artificial intelligence has shifted from fundamental research to practical, real-world applications. GPT-3.5, a model arising from this effort, demonstrated remarkable abilities based on its advanced architecture. Nonetheless, its successor, GPT-4, surpassed it, showing a 40% improvement in performance and showcasing much greater diversity in potential applications.

We can witness a noticeable change in clinical documentation procedures, with the support of AI tools becoming increasingly evident. By systematically analyzing extensive medical datasets, AI models can provide clinicians with evidence-based recommendations, potentially decreasing medical errors and enhancing the effectiveness of treatments.

As telemedicine becomes a vital component, especially in response to recent global health challenges, AI tools with real-time analytical capabilities can enhance patient care. Moreover, in the ever-evolving field of medical academia, where ongoing knowledge updates are crucial, GPT-3.5 and GPT-4 can serve as invaluable, continuously updated references. The incorporation of models like CLIP for the analysis of medical images further emphasizes the potential for enhanced diagnostic accuracy.

The domain of ophthalmology has experienced a considerable rise in research initiatives and the quantity of published research, particularly focusing on the application of AI tools in this branch of science [2]. In the field of ophthalmology, Deep Learning (DL) has been utilized for the analysis of fundus photographs, optical coherence tomography, and visual fields. It has demonstrated strong classification capabilities in identifying conditions such as diabetic retinopathy, retinopathy of prematurity, glaucoma-like disc, macular edema, and age-related macular degeneration [2]. In tasks like identifying and grading diabetic retinopathy AI systems have exhibited performance levels that are on par with or, in some cases, superior to experienced ophthalmologists. Nevertheless, despite these promising outcomes, only a limited number of AI systems have been put into practical use within clinical settings [3].

Employing questions sourced from the National Specialty Exam (NSE) is a viable approach for evaluating ChatGPT’s proficiency in retrieving and analyzing specialized information within the ophthalmology field. Our research aimed to evaluate ChatGPT’s performance in responding to queries and to analyze its strengths and weaknesses in comparison to human comprehension and reasoning.

Table 1 illustrates the proportion of correct responses among the language models. GPT-4 achieved a correct response rate of 63.3% (62 out of 98 questions), whereas GPT-3.5 achieved a correct response rate of 37.8% (37 out of 98 questions). Unlike GPT-3.5, GPT-4 reached the passing rate for the NSE. There was a statistically significant (p<0.001) difference between dependent proportions (25,51% (95% CI: 11,97% to 39,05%)). The probability (p1) that ChatGPT-4 will provide a correct response when GPT-3.5 gives an incorrect one is 55.74% (CI_p1= 43.32%–67.5%). The probability (p2) that GPT-4 will provide a correct response when GPT-3.5 also gives a correct response is 75.68% (CI_p2= 59.90% – 86.64%). Cohen’s Kappa is equal to 0.176 (95% CI: 0.009 to 0.343), indicating weak agreement between GPT-4 and GPT-3.5.

Tables 2–6 display LLM performance categorized by question category. Statistically significant (p=0.0433) difference between dependent proportions (18.6% (95% CI: −2.1% to 39.31%)) was observed solely in the Clinical & Case question category meaning that fraction of correct answers is different between two GPTs. The highest agreement between models is observed for questions from the Clinical & Case group (Cohen’s k = 0.458; 95% CI: 0.214 to 0.702).

Tables 7 and 8 display the performance of LLMs based on their confidence level. The probability (p3) of providing a correct answer when the confidence level is “definitely sure” is 43.9% (95% CI: 29.9% – 59%) for GPT-3.5 and 88.9% (95% CI: 56.5% – 98%) for GPT-4. The probability (p4) of giving the correct answer when the confidence level is other than “definitely sure” is 33.3% (95% CI: 22.5% – 46.3%) for GPT-3.5 and 60.7% (95% CI: 50.3% – 70.2%) for GPT-4.

Table 9 displays the proportion of certainty levels assigned by LLMs. The number of observed agreements was 43, accounting for 43.88% of the observations. Cohen’s Kappa coefficient was calculated to be 0.082 (95% CI: −0.018 to 0.182) meaning very week agreement between models.

When comparing the accuracy of the answers using both GPT-3.5 and GPT-4.0 by question category, we discovered that the fields of surgery and pediatrics were the most lacking in terms of answer correctness. The lower accuracy in answering questions in the field of surgery and pediatric ophthalmology could be attributed to several factors, which is important to consider when analyzing the performance of language models like GPT-3.5 and GPT-4.0 in specific domains.

If the training data used for fine-tuning the models did not adequately cover pediatric ophthalmology and surgery, the models might not have gained a robust understanding of the intricacies and nuances specific to these fields, which resulted in a lesser understanding of the question asked and ultimately – an incorrect answer. Pediatric ophthalmology and surgery are highly specialized fields with complex and detailed knowledge requirements, which, if the training data did not sufficiently capture the diversity of cases and scenarios within, might be the cause of the model’s struggles to provide accurate answers. Another aspect worth noting is the question of complex decision-making processes within the NSE questions in the field of surgery. If the models lack exposure to a diverse set of surgical cases and the associated decision-making considerations, their accuracy in these scenarios may be compromised.

The matter of considerable significance also lies in the elevated specialization of medical terminology, particularly within the realm of surgery. This domain is characterized by a high degree of nuance, frequently encompassing intricate and technical language. It is plausible that the models were not specifically fine-tuned to adequately account for such nuanced linguistic intricacies in the context of surgery. This facet, although deliberated within the scope of surgical inquiries, may be extrapolated to encompass all queries embedded in the NSE, consequently yielding test outcomes lower than anticipated.

In Moshirfar et al.’s study, GPT-4 outperformed both GPT-3.5 and humans on ophthalmology questions sourced from a professional-level question bank. GPT-4 performed significantly better than GPT-3.5 (73.2% vs. 55.46%). It is important to note that in this study the subcategory “lens and cataract” presented a unique challenge for the GPT-4 and GPT-3.5 models [4].

In Cai et al.’s study, Bing Chat (Microsoft), ChatGPT 3.5, and ChatGPT 4.0 (OpenAI) were evaluated using 250 questions from the Basic Science and Clinical Science Self-Assessment Program. While ChatGPT’s training data is up to 2021, Bing Chat utilizes a more up-to-date internet search to produce its responses. This was measured against the performance of human participants. Human participants achieved an accuracy rate of 72.2%. ChatGPT-3.5 had the lowest accuracy at 58.8%, while both ChatGPT-4.0, with 71.6%, and Bing Chat, with 71.2%, showed similar performance levels [5].

Integrating AI language models like GPT into healthcare is a complex task with significant challenges that may take time to overcome. Ensuring data privacy is crucial, as medical information is highly sensitive and must adhere to strict regulations. Additionally, the quality and specificity of training data are vital, yet most AI models are trained on datasets not tailored for specialized medical use, which often excludes critical medical knowledge due to accessibility restrictions.

The rapid evolution of medical knowledge presents a unique challenge for AI. Research, treatments, and protocols are continually updated, but AI models like GPT are typically trained on datasets that might not include the latest medical information. This lag can lead to outdated or incorrect recommendations from the AI, which can have serious implications in a healthcare setting. Moreover, a lot of essential clinical data is sequestered behind paywalls or restricted to certain legal and ethical guidelines, limiting the availability of the most current information for AI training.

Ethical concerns are paramount, especially regarding patient consent for using their data, the role of AI in critical decision-making, and ensuring transparency in AI-driven processes. Despite the potential benefits, the application of AI like GPT in healthcare involves navigating technical, regulatory, ethical, and societal hurdles. The journey towards integrating AI effectively in healthcare necessitates ongoing research and careful consideration of its broad impacts.

Comparison of ChatGPT 3.5 and GPT-4.0 in solving a Neurosurgery Specialization Examination reveals several limitations. The reproducibility of results could be affected by the inherent variability in generating responses, even when inputs are identical. Furthermore, NSE consist solely of closed, single-choice questions, which are not necessarily indicative of the complexities encountered in real-world clinical scenarios. The evaluation criteria and metrics currently used fail to capture the intricate nuances of medical reasoning and decision-making inherent in clinical practice, indicating that these tools cannot yet be autonomously integrated into medical practice. Additionally, potential biases in the datasets used for training these models may affect their performance and generalizability across diverse clinical scenarios. Physicians need to remain aware of ongoing AI research, as its application has the potential to elevate future standards of medical care, despite currently existing limitations and ethical concerns related to the use of ChatGPT in clinical practice.