Primary study · included by 1 review
Accuracy of ChatGPT-Generated Information on Head and Neck and Oromaxillofacial Surgery: A Multicenter Collaborative Analysis.
Vaira 2023 · Vaira Luigi Angelo, Lechien Jerome R, Abbate Vincenzo, et al. · Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery · 2024
Included by
| Outcome this study informs | Pooled effect | Evidence | Certainty |
|---|---|---|---|
Accuracy (proportion of correct answers) Any large language model vs Reference standard (official answer key, validated question bank or expert consensus) · Questions answered by LLMs (studies searched to January 2025) | Proportion, percent (random effects, Freeman-Tukey double arcsine) 63.69 [60.26, 67.06] | 39 studies · 17021 participants | GRADE not reported |
Accuracy (proportion of correct answers) Any large language model vs Reference standard (official answer key, validated question bank or expert consensus) · Questions answered by LLMs (studies searched to January 2025) | Leave-one-out analysis demonstrated stable pooled accuracy estimates ranging from 64.13% to 65.66%, with the most influential study (Chau et al) altering results by 1.96% upon exclusion | NR | GRADE not reported |
Accuracy (proportion of correct answers) Any large language model vs Reference standard (official answer key, validated question bank or expert consensus) · Questions answered by LLMs (studies searched to January 2025) | Proportion, percent (leave-one-out) 63.47 [62.69, 64.25] | NR | GRADE not reported |
Accuracy (proportion of correct answers) Any large language model vs Reference standard (official answer key, validated question bank or expert consensus) · Questions answered by LLMs (studies searched to January 2025) | Prediction interval [29.65; 91.71] | 39 studies · 17021 participants | GRADE not reported |
Accuracy (proportion of correct answers) Any large language model vs Reference standard (official answer key, validated question bank or expert consensus) · Questions answered by LLMs (studies searched to January 2025) | Proportion, percent (leave-one-out) 65.04 [64.26, 65.81] | NR | GRADE not reported |
Accuracy (proportion of correct answers) Any large language model vs Reference standard (official answer key, validated question bank or expert consensus) · Questions answered by LLMs (studies searched to January 2025) | Proportion, percent (fixed effect) 64.87 [64.13, 65.61] | 39 studies · 17021 participants | GRADE not reported |
Accuracy (proportion of correct answers) Any large language model vs Reference standard (official answer key, validated question bank or expert consensus) · Questions answered by LLMs (studies searched to January 2025) | Proportion, percent (leave-one-out) 66.83 [66.07, 67.60] | NR | GRADE not reported |
Accuracy (proportion of correct answers) Any large language model vs Reference standard (official answer key, validated question bank or expert consensus) · Questions answered by LLMs (studies searched to January 2025) | Proportion, percent (random effects, Freeman-Tukey double arcsine) 65.43 [61.88, 68.89] | NR | GRADE not reported |
1 including review in total.