Evaluation of the responses from different chatbots to frequently asked patient questions about impacted canines
und
01. Sept. 2025
Über diesen Artikel
Online veröffentlicht: 01. Sept. 2025
Seitenbereich: 288 - 300
Eingereicht: 01. März 2025
Akzeptiert: 01. Mai 2025
DOI: https://doi.org/10.2478/aoj-2025-0020
Schlüsselwörter
© 2025 Elif Gökçe Erkan Acar et al., published by Sciendo
This work is licensed under the Creative Commons Attribution 4.0 International License.
Figure 1.
Questions used in the study
| Questions | |
|---|---|
| 1 | What is impacted tooth? |
| 2 | What are the most common impacted tooth in humans? |
| 3 | What is the frequency of canine impaction? |
| 4 | What is the most common cause of canine impaction? |
| 5 | What are the signs of impacted canine tooth? |
| 6 | Do impacted canines have to be removed? |
| 7 | What age do you treat impacted canine? |
| 8 | Can an impacted canine come down on its own? |
| 9 | What happens if impacted canine tooth is left untreated ? |
| 10 | If impacted canine is left untreated, what would be the side effects? |
| 11 | Who does the treatment for impacted canine teeth? |
| 12 | How is the decision made whether to extract or maintain an impacted tooth? |
| 13 | Should I have orthodontic treatment to treat impacted canine? |
| 14 | How does the orthodontic treatment for impacted canine is done? |
| 15 | What problems can occur during the treatment of impacted canine? |
| 16 | Does treatment of impacted canine can be done with braces? |
| 17 | Does treatment of impacted canine can be done with aligners? |
| 18 | Can impacted canine tooth erupt on its own? |
| 19 | Can impacted canine tooth erupt with jaw expansion? |
| 20 | Can all impacted canine teeth be saved with orthodontic treatment? |
| 21 | How long does it take to erupt an impacted canine? |
| 22 | Does orthodontic treatment of impacted canine tooth cause pain? |
| 23 | Is x-ray necessary for impacted canine tooth treatment? |
| 24 | How is impacted canine tooth exposure surgery done? |
| 25 | Who does the canine tooth exposure surgery? |
| 26 | What is closed technique for impacted canine? |
| 27 | What is gold chain in impacted canine treatment? |
| 28 | How long does it take to recover from impacted canine exposure surgery? |
| 29 | How is oral care performed after the canine exposure surgery? |
| 30 | Does impacted tooth exposure surgery cause pain? |
| 31 | When can I return to work and school after impacted canine exposure surgery? |
| 32 | Does orthodontic treatment to bring an impacted canine into position cause damage to the tooth? |
| 33 | Does another tooth regrow after treatment of impacted tooth? |
| 34 | Will my teeth shift after the orthodontic treatment of an impacted canine tooth? |
| 35 | How is retention done after impacted canine tooth treatment? |
Modified DISCERN, Likert and AOI index scores of ChatGPT, Claude and Gemini
| ChatGPT | Claude | Gemini | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Mean ± SD | Median (Min-Max) | Mean ± SD | Median (Min-Max) | Mean ± SD | Median (Min-Max) | ||||
| Modified Discern Score | 28.13 ± 2.83 | 29.00 (20.00-32.00) | 29.70 ± 3.08 | 31.00 (23.00-33.00) | 33.66 ± 2.64 | 34.00 (28.00-38.00) | ChatGPT vs Claude | ChatGPT vs Gemini | Claude vs Gemini |
| 0.385 | 0.028 |
0.011 |
|||||||
| Likert Score | 4.76 ± 0.43 | 5.00 (4.00-5.00) | 4.71 ±0.47 | 5.00 (3.50-5.00) | 4.66 ± 0.47 | 5.00 (4.00-5.00) | 0.61 | ||
| AOI Index | 8.67 ±0.55 | 9.00 (7.00-9.00) | 8.37 ± 0.78 | 9.00 (6.00-9.00) | 8.1 1 ± 0.84 | 8.00 (6.00-10.00) | ChatGPT vs Claude | ChatGPT vs Gemini | Claude vs Gemini |
| 0.042 |
0.036 |
0.360 | |||||||
Flesch-Kincaid Readilibity Scores, Flesch-Kincaid Grade Levels and word counts of ChatGPT, Claude and Gemini
| ChatGPT | Claude | Gemini | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Mean ± SD | Median (Min-Max) | Mean ± SD | Median (Min-Max) | Mean ± SD | Median (Min-Max) | ||||
| Flesch-Kincaid Reading Ease Score | 37.89 ± 8.08 | 36.90 (14.80-51.00) | 35.46 ± 10.47 | 36.30 (5.70-51.10) | 40.19 ± 9.85 | 38.40 (18.30-62.40) | 0.121 | ||
| Flesch-Kincaid Grade Level | 12.99 ± 1.70 | 13.30 (9.70-17.50) | l2.64 ± 1.39 | 12.40 (10.40-15.90) | 12.44 ± 1.83 | 12.60 (8.40-15.90) | 0.377 | ||
| Word Count | 239.74 ± 114.21 | 237.00 (64.00-490.00) | 185.89 ±49.91 | 181.00 (95.00-285.00) | 219.29 ± 96.54 | 187.00 (65.00-430.00) | ChatGPT vs Claude | Claude vs. Gemini | ChatGPT vs. Gemini |
| 0.019 |
0.001 |
0.041 |
|||||||
The Modified DISCERN, Likert Scoring, AOI Index, FRES and FKGL
| MODIFIED DISCERN | ||
|---|---|---|
| Questions | Scoring (Minimum-maximum) | |
| 1. Are the aims clear? | 1-5 | |
| 2. Does it achieve its aims? | 1-5 | |
| 3. Is it relevant? | 1-5 | |
| 4. Is it clear what sources of information were used to compile the publication? | 1-5 | |
| 5. Is it clear when the information used or reported in the publication was produced? | 1-5 | |
| 6. Is it balanced and unbiased? | 1-5 | |
| 7. Does it provide details of additional sources of support and information? | 1-5 | |
| 8. Does it refer to areas of uncertanity? | 1-5 | |
| Total Modified DİSCERN Score | 8-40 (Poor (8–15 point), moderate (16–31 point), or good (32–40 point)) | |
| LIKERT Scale | ||
| Parameter | Score | |
| The chatbot’s answers are completely incorrect | 1 | |
| The chatbot’s answers contain more incorrect items than correct items | 2 | |
| The chatbot’s answers contain an equal balance of correct and incorrect items | 3 | |
| The chatbot’s answers contain more correct items than incorrect items | 4 | |
| The chatbot’s answers are completely correct | 5 | |
| AOI INDEX | ||
| Item | Definition | Score (minimum-maximum) |
| Factual accuracy | The response aligns with known facts, data, or established knowledge on the subject | 0-2 |
| Corroboration | The response is based on evidence from textbooks, studies, or guidelines | 0-2 |
| Consistency | The response is internally consistent and does not contain contradictory 2 statements | 0-2 |
| Clarity and specificity | The response is clear and specific, avoiding vague or ambiguous language | 0-2 |
| Relevance | The response directly addresses and adheres to the question or topic posed | 0-2 |
| Total AOI score | The sum of all scores | 0-10 |
| FLESCH KINCAID READABILITY | ||
| Flesh Reading Ease Score | Grade Level | Reading level |
| 90-100 | 5 | Very Easy |
| 80-90 | 6 | Easy |
| 70-80 | 7 | Fairly Easy |
| 60-70 | 8-9 | Standard and/or Plain |
| 50-60 | 10-12 | Fairly Difficult |
| 30-50 | College | Difficult |
| 0-30 | College Grad | Very Difficult |