Evaluation of the responses from different chatbots to frequently asked patient questions about impacted canines

Figure 1.

Questions used in the study

	Questions
1	What is impacted tooth?
2	What are the most common impacted tooth in humans?
3	What is the frequency of canine impaction?
4	What is the most common cause of canine impaction?
5	What are the signs of impacted canine tooth?
6	Do impacted canines have to be removed?
7	What age do you treat impacted canine?
8	Can an impacted canine come down on its own?
9	What happens if impacted canine tooth is left untreated ?
10	If impacted canine is left untreated, what would be the side effects?
11	Who does the treatment for impacted canine teeth?
12	How is the decision made whether to extract or maintain an impacted tooth?
13	Should I have orthodontic treatment to treat impacted canine?
14	How does the orthodontic treatment for impacted canine is done?
15	What problems can occur during the treatment of impacted canine?
16	Does treatment of impacted canine can be done with braces?
17	Does treatment of impacted canine can be done with aligners?
18	Can impacted canine tooth erupt on its own?
19	Can impacted canine tooth erupt with jaw expansion?
20	Can all impacted canine teeth be saved with orthodontic treatment?
21	How long does it take to erupt an impacted canine?
22	Does orthodontic treatment of impacted canine tooth cause pain?
23	Is x-ray necessary for impacted canine tooth treatment?
24	How is impacted canine tooth exposure surgery done?
25	Who does the canine tooth exposure surgery?
26	What is closed technique for impacted canine?
27	What is gold chain in impacted canine treatment?
28	How long does it take to recover from impacted canine exposure surgery?
29	How is oral care performed after the canine exposure surgery?
30	Does impacted tooth exposure surgery cause pain?
31	When can I return to work and school after impacted canine exposure surgery?
32	Does orthodontic treatment to bring an impacted canine into position cause damage to the tooth?
33	Does another tooth regrow after treatment of impacted tooth?
34	Will my teeth shift after the orthodontic treatment of an impacted canine tooth?
35	How is retention done after impacted canine tooth treatment?

Modified DISCERN, Likert and AOI index scores of ChatGPT, Claude and Gemini

	ChatGPT		Claude		Gemini
	Mean ± SD	Median (Min-Max)	Mean ± SD	Median (Min-Max)	Mean ± SD	Median (Min-Max)		P value
Modified Discern Score	28.13 ± 2.83	29.00 (20.00-32.00)	29.70 ± 3.08	31.00 (23.00-33.00)	33.66 ± 2.64	34.00 (28.00-38.00)	ChatGPT vs Claude	ChatGPT vs Gemini	Claude vs Gemini
							0.385	0.028*	0.011*
Likert Score	4.76 ± 0.43	5.00 (4.00-5.00)	4.71 ±0.47	5.00 (3.50-5.00)	4.66 ± 0.47	5.00 (4.00-5.00)		0.61
AOI Index	8.67 ±0.55	9.00 (7.00-9.00)	8.37 ± 0.78	9.00 (6.00-9.00)	8.1 1 ± 0.84	8.00 (6.00-10.00)	ChatGPT vs Claude	ChatGPT vs Gemini	Claude vs Gemini
							0.042*	0.036*	0.360

Flesch-Kincaid Readilibity Scores, Flesch-Kincaid Grade Levels and word counts of ChatGPT, Claude and Gemini

	ChatGPT		Claude		Gemini
	Mean ± SD	Median (Min-Max)	Mean ± SD	Median (Min-Max)	Mean ± SD	Median (Min-Max)		P value
Flesch-Kincaid Reading Ease Score	37.89 ± 8.08	36.90 (14.80-51.00)	35.46 ± 10.47	36.30 (5.70-51.10)	40.19 ± 9.85	38.40 (18.30-62.40)	0.121
Flesch-Kincaid Grade Level	12.99 ± 1.70	13.30 (9.70-17.50)	l2.64 ± 1.39	12.40 (10.40-15.90)	12.44 ± 1.83	12.60 (8.40-15.90)	0.377
Word Count	239.74 ± 114.21	237.00 (64.00-490.00)	185.89 ±49.91	181.00 (95.00-285.00)	219.29 ± 96.54	187.00 (65.00-430.00)	ChatGPT vs Claude	Claude vs. Gemini	ChatGPT vs. Gemini
							0.019*	0.001*	0.041*

The Modified DISCERN, Likert Scoring, AOI Index, FRES and FKGL

MODIFIED DISCERN
Questions	Scoring (Minimum-maximum)
1. Are the aims clear?	1-5
2. Does it achieve its aims?	1-5
3. Is it relevant?	1-5
4. Is it clear what sources of information were used to compile the publication?	1-5
5. Is it clear when the information used or reported in the publication was produced?	1-5
6. Is it balanced and unbiased?	1-5
7. Does it provide details of additional sources of support and information?	1-5
8. Does it refer to areas of uncertanity?	1-5
Total Modified DİSCERN Score	8-40 (Poor (8–15 point), moderate (16–31 point), or good (32–40 point))
LIKERT Scale
Parameter	Score
The chatbot’s answers are completely incorrect	1
The chatbot’s answers contain more incorrect items than correct items	2
The chatbot’s answers contain an equal balance of correct and incorrect items	3
The chatbot’s answers contain more correct items than incorrect items	4
The chatbot’s answers are completely correct	5
AOI INDEX
Item	Definition	Score (minimum-maximum)
Factual accuracy	The response aligns with known facts, data, or established knowledge on the subject	0-2
Corroboration	The response is based on evidence from textbooks, studies, or guidelines	0-2
Consistency	The response is internally consistent and does not contain contradictory 2 statements	0-2
Clarity and specificity	The response is clear and specific, avoiding vague or ambiguous language	0-2
Relevance	The response directly addresses and adheres to the question or topic posed	0-2
Total AOI score	The sum of all scores	0-10
FLESCH KINCAID READABILITY
Flesh Reading Ease Score	Grade Level	Reading level
90-100	5	Very Easy
80-90	6	Easy
70-80	7	Fairly Easy
60-70	8-9	Standard and/or Plain
50-60	10-12	Fairly Difficult
30-50	College	Difficult
0-30	College Grad	Very Difficult