An evaluation of orthodontic information quality regarding artificial intelligence (AI) chatbot technologies: A comparison of ChatGPT and google BARD

Artificial Intelligence (AI) may be defined as a sequence of actions to achieve a particular objective.¹ As a sub-field of computer science, it holds the potential to replicate human intelligence to make predictions and complex decisions.^1,2 AI has been increasingly integrated into various modern services, such as content recommendation, facial recognition and smart speakers, and so has created many exciting opportunities.³

In the medical and dental fields, substantial progress has been made in utilising Artificial Intelligence, including various technologies related to medical image analysis, data mining and natural language processing. The development of high-performance computing has made it feasible to extract and obtain essential data from large datasets. This data extraction process is commonly called machine learning (ML).⁴

The utilisation of chatbots, such as ChatGPT (Chat Generative Pre-trained Transformer) developed by OpenAI (OpenAI, San Francisco, CA, USA), can significantly enhance healthcare delivery through simulated natural conversation with human users. ChatGPT, released in November 2022, is currently the most prominent chatbot in the medical field. Additionally, Google (Google, Mountain View, CA, USA) has unveiled a chatbot called BARD, which shares similarities with ChatGPT but boasts real-time internet access and employs the Language Model for Dialogue Applications (LaMDA) as its communication model.¹

In the context of medical practice, chatbots have a significant influence on the healthcare system.⁵ They can be a complementary tool for diagnosing and making decisions in medical fields, including cardiology, radiology, and urology. Sophisticated language models, such as ChatGPT, also possess substantial potential for enhancing clinical applications and research in dentistry. Their prudent use can bring about a transformative change in dental diagnosis and treatment planning. Moreover, integrating diverse medical examination data in investigations can help achieve precision and personalised dentistry and medicine objectives. However, it is crucial to acknowledge the inherent restrictions and potential risks associated with implementing AI technologies. Therefore, the present study aimed to assess the accuracy of the answers provided by ChatGPT and BARD, the most widely used chatbot applications, when questions related to orthodontics were posed. The null hypothesis was that there is no difference in the quality of the answers provided by both tools.

Based on the study conducted by Perez-Pino et al.,⁶ a power analysis⁷ determined that a minimum of 20 questions were required for an 80% power test. After clearing all past searches, Google was interrogated to gather the most frequently asked questions about orthodontic treatment, and the questions from the first 50 pages were analysed individually. Of the popular queries about conventional braces, clear aligners, orthognathic surgery, and orthodontic retainers, a total of 24 questions were selected that had been repeated several times (Table I). The answers given by chatGPT and google BARD to an example question are shown in Table II.

An experienced orthodontist (CA, Investigator 1) and an orthodontic resident (KK, Investigator 2) interrogated the two systems and rated their responses using a five-point modified Likert scale: (5) the system responded with evidence-based information; (4) the system responded with adequate information; (3) the system responded but did not provide adequate information; (2) the system responded with the wrong answer; and (1) the system did not know the response to the question. The Likert scale ratings were designed to reflect the utility and efficiency of the systems, with (1) being the worst rating. The results of the queries were recorded on Microsoft Excel (Microsoft Corp., New Mexico, USA) for comparative evaluation. The number of words and listed references in the answers were also recorded (Tables I).

When inter-observer reliability was analysed, a statistically significant and moderate correlation was found between the scores of the two investigators for ChatGPT (ICC = 0.739; p = 0.733). Similarly, a statistically significant and solid correlation was found between the BARD scores (ICC =0.784; p = 0.733) (Table III).

An examination of the relationships between the word counts, reference counts, and ChatGPT scores revealed no statistically significant correlation between Investigator 1’s ChatGPT scores and the ChatGPT word counts (p = 0.113). However, a statistically significant and moderate relationship was observed between Investigator 1’s ChatGPT scores and the number of ChatGPT listed references (r = 0.685, p < 0.001). Additionally, no statistically significant relationship was found between the number of ChatGPT words and the number of references (p = 0.266) (Table IV). Similarly, Investigator 2’s ChatGPT scores were not found to have a statistically significant relationship with the number of ChatGPT words (P = 0.536). However, a statistically significant and close correlation was observed between Investigator 2’s ChatGPT scores and the number of ChatGPT references (r = 0.708, p < 0.001), and no statistically significant relationship was found between the number of ChatGPT words and the number of references (p = 0.266) (Table V).

In analysing the relationships between word counts, listed reference counts, and Google BARD scores, no significant association was identified between Investigator 1’s BARD score and BARD word counts (p = 0.120). However, a moderate and statistically significant relationship was found between Investigator 1’s BARD score and the number of references (r = 0.621; p = 0.001). Additionally, no significant association was observed between the number of BARD words and the number of references (p = 0.889) (Table VI). For Investigator 2, no significant relationship was discovered between the BARD score and the number of BARD words (p = 0.382). Instead, a moderate and statistically significant association was found between Investigator 2’s BARD score and the number of BARD listed references (r = 0.691; p < 0.001). No significant relationship was noted between the number of BARD words and the number of references (p = 0.889) (Table VII).

A statistically significant difference was found in both investigators’ numerical rating scales based on the tools used (p = 0.014 and p = 0.030, respectively). The rank mean value was 29.15 in ChatGPT and 19.85 in BARD for Investigator 1. Similarly, the rank mean value was 28.63 in the ChatGPT and 20.38 in BARD for Investigator 2. However, there was no statistically significant difference in the mean word count (p = 0.732) or number of references (p = 0.796) between the two groups (Table VIII).

AI is a branch of computer science that aims to replicate human intelligence in computer systems. Recent advancements have enabled the automation of tasks once thought to be exclusively performed by humans.⁶ Generative AI is one such implementation area, and chatbots are a prominent example. The question of whether computers can think has long been a topic of interest, and chatbots have been at the centre of this debate.⁸ Leading companies such as Google, Meta, DeepMind, and OpenAI are actively developing chatbots. The popularity of chatbots has recently increased following OpenAI’s decision to make its language models and user interfaces publicly available through ChatGPT.

ChatGPT is a sophisticated natural language processing model that has been pre-trained on a vast dataset to enable the generatation of language that is coherent, contextually appropriate, and often indistinguishable from human-written text. This model utilises a transformer architecture that combines the advantages of generative models with scalability and employs a self-attention mechanism to learn the context of the dialogue and provide more accurate replies.⁹ BARD AI is a cloud-based conversational AI platform powered by LaMDA which is a collection of conversational neural language models developed by Google. This platform allows organisations to build and deploy chatbots that communicate with consumers through various channels, such as websites, messaging applications, and voice assistants. BARD AI has been used and will continue to be used in a number of fields, including academic writing, healthcare, education, and law, among others.¹⁰ Although the reliability of these environments is widely debated, to date, there are no studies identified that evaluate the reliability of BARD and ChatGPT in the field of orthodontics.

Upon evaluation of the results of the present study, it is apparent that ChatGPT outperforms BARD in providing higher-quality answers to popular orthodontic questions posed by patients. This finding aligns with the those of Rahsepar et al.¹⁰ who assessed the reliability of ChatGPT and BARD in answering common lung cancer questions. Additionally, recent studies suggest that ChatGPT surpasses BARD in answering cancer-related questions and neurosurgery oral board preparation questions, a conclusion that is consistent with the present analysis.¹¹ The superior performance of ChatGPT can be attributed to the variation in their respective training data and the strength of their models in addressing medical enquiries.^10,11 Previous studies have reported that Google BARD provides answers with more words, whereas ChatGPT offers better quality responses.^10–12 However, in the present study, it was noted that both applications offered answers containing a similar number of words. The higher reliability of ChatGPT was attributed to the provision of answers with more references, as evidence-based information is considered more reliable and acceptable by scientists.

In March 2023, OpenAI introduced the latest version of ChatGPT, GPT 4. Research has demonstrated that GPT 4 outperforms GPT 3.5 in addressing medical queries.^5,13 This superiority may be attributed to GPT 4’s increased access to current resources. However, in the present study, GPT 3.5 was chosen due to the belief that free chatbot applications, such as GPT 3.5 and Google BARD, will be more appealing to patients seeking online information about orthodontics compared to paid applications.

Despite the recent and significant advancements in chatbot applications such as ChatGPT and BARD, their use in the health field remains limited without professional support. A recent study aimed to compare the performance of orthopaedic residents, ChatGPT 3.5 and GPT 4, on orthopaedic assessment examinations.¹⁴ It was reported that the residents outperformed ChatGPT 3.5 and GPT 4 on the assessment examinations, and it was therefore concluded that it is unlikely that GPT 4 or ChatGPT 3.5 would pass the American Board of Orthopaedic Surgery written examination. While the answers provided by ChatGPT and BARD applications in the present study were generally of good quality, it should be noted that patients posed these questions. Therefore, it is not surprising that more superficial answers were acceptable in the study. The quality of answers provided by chatbot applications, when asked to answer orthodontic questions at the graduate level, remains an open topic for discussion.

More research is needed to comprehensively assess the capabilities and constraints of these models and determine the most effective methods for their incorporation into clinical practice. While AI language models may become more prevalent, healthcare professionals remain responsible for ultimately making decisions regarding diagnoses, investigations, and treatments.