Performance of ChatGPT and GPT-4 on Polish National Specialty Exam (NSE) in Ophthalmology
Marcin Ciekalski
Orcid profile, Maciej Laskowski
Orcid profile, Agnieszka Koperczak
Orcid profile, Maria Śmierciak
Orcid profile and Sebastian Sirek
Orcid profile 
Article Category: Original Study
Published Online: Sep 23, 2024
Page range: 111 - 116
Received: Jan 11, 2024
Accepted: Jun 19, 2024
DOI: https://doi.org/10.2478/ahem-2024-0006
Keywords
© 2024 Marcin Ciekalski et al., published by Sciendo
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Introduction
Artificial intelligence (AI) has evolved significantly, driven by advancements in computing power and big data. Technologies like machine learning and deep learning have led to sophisticated models such as GPT-3.5 and GPT-4. This study assesses the performance of these AI models on the Polish National Specialty Exam in ophthalmology, exploring their potential to support research, education, and clinical decision-making in healthcare.
Materials and Methods
The study analyzed 98 questions from the Spring 2023 Polish NSE in Ophthalmology. Questions were categorized into five groups: Physiology & Diagnostics, Clinical & Case Questions, Treatment & Pharmacology, Surgery, and Pediatrics. GPT-3.5 and GPT-4 were tested for their accuracy in answering these questions, with a confidence rating from 1 to 5 assigned to each response. Statistical analyses, including the Chi-squared test and Mann-Whitney U test, were employed to compare the models’ performance.
Results
GPT-4 demonstrated a significant improvement over GPT-3.5, correctly answering 63.3% of questions compared to GPT-3.5’s 37.8%. GPT-4’s performance met the passing criteria for the NSE. The models showed varying degrees of accuracy across different categories, with a notable gap in fields like surgery and pediatrics.
Conclusions
The study highlights the potential of GPT models in aiding clinical decisions and educational purposes in ophthalmology. However, it also underscores the models’ limitations, particularly in specialized fields like surgery and pediatrics. The findings suggest that while AI models like GPT-3.5 and GPT-4 can significantly assist in the medical field, they require further development and fine-tuning to address specific challenges in various medical domains.