An evaluation of orthodontic information quality regarding artificial intelligence (AI) chatbot technologies: A comparison of ChatGPT and google BARD
, , et | 03 juin 2024
À propos de cet article
Publié en ligne: 03 juin 2024
Pages: 149 - 157
Reçu: 01 janv. 2024
Accepté: 01 mars 2024
DOI: https://doi.org/10.2478/aoj-2024-0012
© 2024 Can Arslan et al., published by Sciendo
This work is licensed under the Creative Commons Attribution 4.0 International License.
An analysis of the relationship between the variables (word and reference counts) with Investigator 1’s rating scale for ChatGPT
| Investigator 1’s ChatGPT scores (Mean) | Number of words (ChatGPT) (Mean) | ||
|---|---|---|---|
| Number | 0.332 | ||
| of words | 0.113 | ||
| (ChatGPT) | |||
| Number of | 0.685 | 0.237 | |
| references | <0.001 |
0.266 | |
| (ChatGPT) |
Comparison of the numerical rating scale, word and reference count values according to the tool used
| Tool | ||||||
|---|---|---|---|---|---|---|
| ChatGPT | BARD | Mann Whitney |
||||
| Mean ± SD | Median (Min–Max) | Mean ± SD | Median (Min–Max) | |||
| Investigator 1 | 5.17 ± 0.76 | 5 (4–6) | 4.63 ±0.65 | 5 (4–6) | 0.014 |
|
| Investigator 2 | 5.17 ± 0.87 | 5 (3 - 6) | 4.67 ± 0.76 | 5 (3 - 6) | U = 189.000 | 0.030 |
| Number of Words | 337.04 ± 82.81 | 349.5 (125 - 464) | 328.46 ± 89.89 | 321.5 (189 - 551) | t = 0.344 | 0.732 |
| Number of References | 2.13 ± 1.51 | 2.5 (0 - 6) | 1.96 ± 1.76 | 2.5 (0 - 5) | U = 276.000 | 0.796 |
An analysis of the relationship between the variables (word and reference counts) with Investigator 2’s rating scale for ChatGPT
| Investigator 2’s ChatGPT scores (Mean) | Number of words (ChatGPT) (Mean) | ||
|---|---|---|---|
| Number | 0.133 | ||
| of words | 0.536 | ||
| (ChatGPT) | |||
| Number of | 0.708 | 0.237 | |
| references | <0.001 |
0.266 | |
| (ChatGPT) |
An analysis of the relationship between the variables (word and listed reference counts) with Investigator 2’s rating scale for BARD
| Investigator 2’s BARD Scores (Mean) | Number of Words (BARD) (Mean) | ||
|---|---|---|---|
| Number of | -0.187 | ||
| words (BARD) | 0.382 | ||
| Number of | 0.691 | -0.030 | |
| references | <0.001 |
0.889 | |
| (BARD) |
Descriptive statistics of the questions asked in the study
| Questions | Investigator 1 (XX) (Mean ± SD) | Investigator 2 (XX) (Mean ± SD) | Number of words (Mean ± SD) | Number of references (Mean ± SD) | |
|---|---|---|---|---|---|
| 1 | What iss the best age to visit the orthodontist? | 5 ± 0 | 5 ± 0 | 240 ± 41.01 | 2 ± 1.41 |
| 2 | Will my braces interfere with my sport activities? | 4.5 ± 0.71 | 4.5 ± 0.71 | 294 ± 48.08 | 0.5 ± 0.71 |
| 3 | What can I eat with braces? | 5 ± 0 | 5 ± 0 | 270 ± 97.58 | 2 ± 1.41 |
| 4 | How do I care for my braces? | 5.5 ± 0.71 | 5 ± 0 | 363.5 ± 94.05 | 3 ± 2.83 |
| 5 | Can an adult do orthodontic treatment? | 5 ± 1.41 | 5 ± 1.41 | 188 ± 89.1 | 2 ± 2.83 |
| 6 | Do the braces interfere with playing an instrument? | 4.5 ± 0.71 | 5.5 ± 0.71 | 381.5 ± 47.38 | 2.5 ± 0.71 |
| 7 | How long does orthodontic treatment take? | 5 ± 1.41 | 5 ± 1.41 | 285 ± 48.08 | 1.5 ± 2.12 |
| 8 | Does orthodontic treatment hurt? | 4.5 ± 0.71 | 4.5 ± 0.71 | 313.5 ± 53.03 | 1 ±1.41 |
| 9 | How to brush your teeth with braces? | 5 ± 0 | 5 ± 0 | 378.5 ± 78.49 | 3.5 ± 0.71 |
| 10 | Do I have to wear orthodontic retainers and how long will I have to wear my retainer? | 4 ± 0 | 3.5 ± 0.71 | 316.5 ± 111.02 | 0 ± 0 |
| 11 | How often will I have to go to the orthodontist? | 4 ± 0 | 4 ± 0 | 279.5 ± 38.89 | 0 ± 0 |
| 12 | How to get orthodontic treatment without metal braces? | 5 ± 1.41 | 4 ± 1.41 | 387.5 ± 53.03 | 4.5 ± 2.12 |
| 13 | What is orthognathic surgery? | 5 ± 0 | 5.5 ± 0.71 | 381 ± 86.27 | 3 ± 0 |
| 14 | How do braces move teeth in orthodontics? | 5.5 ± 0.71 | 5.5 ± 0.71 | 361.5 ± 2.12 | 3 ± 0 |
| 15 | What are the types of braces? | 5 ± 1.41 | 5 ± 1.41 | 407.5 ± 30.41 | 15 ± 2.12 |
| 16 | What is clear aligner treatment? | 5.5 ± 0.71 | 5.5 ± 0.71 | 436.5 ± 113.84 | 4 ± 1.41 |
| 17 | What are the risks and side effects of the orthodontic treatment? | 4.5 ± 0.71 | 4 ± 0 | 507.5 ± 61.52 | 0 ± 0 |
| 18 | What should I do if a bracket comes off or an appliance breaks? | 4.5 ± 0.71 | 5 ± 0 | 373.5 ± 50.2 | 3 ± 0 |
| 19 | What should I do if I have pain or other problems with my braces? | 4.5 ± 0.71 | 4.5 ± 0.71 | 364 ± 1.41 | 1 ± 1.41 |
| 20 | Will wisdom teeth cause teeth to shift? | 5.5 ± 0.71 | 6 ± 0 | 305.5 ± 82.73 | 2.5 ± 0.71 |
| 21 | Which treatment is faster ? Clear aligners or braces? | 5.5 ± 0.71 | 5.5 ± 0.71 | 289.5 ± 142.13 | 2 ± 0 |
| 22 | Can I undergo orthognathic surgery with clear aligners? | 5 ± 1.41 | 5 ± 1.41 | 252.5 ± 26.16 | 2 ± 2.83 |
| 23 | Why do some people need extractions before orthodontic treatment? | 5.5 ± 0.71 | 6 ± 0 | 327.5 ± 12.02 | 3 ± 0 |
| 24 | Can I have orthognathic surgery without orthodontic treatment? | 4.5 ± 0.71 | 4.5 ± 0.71 | 282 ± 38.18 | 1.5 ± 2.12 |
Question and answer format for ChatGPT and BARD
| Answers | Question 1. What’s the best age to visit the orthodontist? |
|---|---|
| ChatGPT | The best age to visit the orthodontist can vary depending on individual circumstances, but there are generally two main stages when people seek orthodontic treatment: Early Treatment (Phase I): This usually occurs between the ages of 7 and 10. At this stage, a child’s adult teeth are beginning to come in, and an orthodontist can identify potential issues with tooth and jaw development. Early treatment may be recommended for problems like crowding, crossbites, or severe overbites. Adolescent/Teen Years (Phase II): This is the most common time for orthodontic treatment. Typically, it begins between the ages of 11 and 15. By this age, most of the adult teeth have erupted, and any existing dental issues are more easily identified. This is also a period when the jaw is still growing, which makes it an ideal time for more complex treatments. Adult Treatment: Orthodontic treatment can be done at any age. Many adults seek orthodontic care to address issues that were not treated during childhood or to correct new problems that have developed over time. |
| Google Bard | The American Association of Orthodontists (AAO) recommends that all children have their first orthodontic evaluation by the age of 7. This is because most permanent teeth have erupted by this age, and an orthodontist can identify any potential problems with the development of the jaw and teeth. Early intervention can help to prevent more serious problems down the road, and it can also make orthodontic treatment shorter and easier. For example, if a child has a crossbite (where the upper jaw overlaps the lower jaw), early treatment can help to correct the problem before it becomes more severe. Of course, not all children will need orthodontic treatment. But having an early evaluation can give parents and orthodontists the opportunity to identify any potential problems and develop a plan to address them. |
Inter-observer reliability scores
| ICC (95% CI) | ||
|---|---|---|
| ChatGPT | 0.739 (0.485–0.878) | <0.001 |
| BARD | 0.784 (0.563–0.9) | <0.001 |
An analysis of the relationship between the variables (word and reference counts) with Investigator 1’s rating scale for BARD
| Investigator 1’s BARD scores (Mean) | Number of words (BARD) (Mean) | ||
|---|---|---|---|
| Number of | -0.326 | ||
| words (BARD) | 0.120 | ||
| Number of | 0.621 | -0.030 | |
| References | 0.001 |
0.889 | |
| (BARD) |