Non-Cochrane · other · PM42150339
Accuracy of Large Language Models in Answering Dental Examination Questions: A Systematic Review and Meta-Analysis.
Dashti Mahmood, Khosraviani Farshad, Meyari Atieh, et al. · International dental journal · 2026 · PubMed · doi:10.1016/j.identj.2026.109609
Findings
Population Dental examination and knowledge question sets
30 findings
| Any large language model | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets | Proportion, percent (random effects, Freeman-Tukey double arcsine) 63.69 [60.26, 67.06] GRADE not reported 39 studies · 17021 participants |
| Any large language model | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · Leave-one-out: omitting Sabri et al. 2024 (ChatGPT 4 estimate) Unresolved: the two extractors disagree on this row. | Proportion, percent (leave-one-out) 63.47 [62.69, 64.25] GRADE not reported Studies not reported |
| Any large language model | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · Leave-one-out: omitting Chau et al. 2023 (ChatGPT 3.5 estimate) | Proportion, percent (leave-one-out) 66.83 [66.07, 67.60] GRADE not reported Studies not reported |
| Any large language model | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · Leave-one-out sensitivity analysis (range stated in text) | Leave-one-out analysis demonstrated stable pooled accuracy estimates ranging from 64.13% to 65.66%, with the most influential study (Chau et al) altering results by 1.96% upon exclusion GRADE not reported Studies not reported |
| Any large language model | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · 95% prediction interval Unresolved: the two extractors disagree on this row. | Prediction interval [29.65; 91.71] GRADE not reported 39 studies · 17021 participants |
| Any large language model | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · Leave-one-out: omitting Chau et al. 2023 (ChatGPT 4 estimate) Unresolved: the two extractors disagree on this row. | Proportion, percent (leave-one-out) 65.04 [64.26, 65.81] GRADE not reported Studies not reported |
| Any large language model | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · Fixed-effect model (sensitivity) | Proportion, percent (fixed effect) 64.87 [64.13, 65.61] GRADE not reported 39 studies · 17021 participants |
| Any large language model | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · Trim-and-fill (6 imputed studies) | Proportion, percent (random effects, Freeman-Tukey double arcsine) 65.43 [61.88, 68.89] GRADE not reported Studies not reported |
| Bard (Gemini in its Bard era) | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · MODEL = Bard; fixed-effect model | Proportion, percent (fixed effect) 59.94 [56.57, 63.27] GRADE not reported 874 participants |
| Bard (Gemini in its Bard era) | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · MODEL = Bard | Proportion, percent (random effects, Freeman-Tukey double arcsine) 59.39 [50.65, 67.86] GRADE not reported 874 participants |
| ChatGPT 3.5 | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · MODEL = ChatGPT 3.5; fixed-effect model | Proportion, percent (fixed effect) 56.63 [55.33, 57.92] GRADE not reported 5877 participants |
| ChatGPT 3.5 | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · MODEL = ChatGPT 3.5 | Proportion, percent (random effects, Freeman-Tukey double arcsine) 60.41 [53.17, 67.44] GRADE not reported 5877 participants |
| Microsoft Bing (Copilot predecessor) | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · MODEL = Bing; fixed-effect model | Proportion, percent (fixed effect) 67.71 [64.35, 70.99] GRADE not reported 827 participants |
| Microsoft Bing (Copilot predecessor) | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · MODEL = Bing | Proportion, percent (random effects, Freeman-Tukey double arcsine) 61.14 [48.73, 72.91] GRADE not reported 827 participants |
| Claude | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · MODEL = Claude | Proportion, percent (random effects, Freeman-Tukey double arcsine) 57.75 [38.09, 76.30] GRADE not reported 216 participants |
| Claude | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · MODEL = Claude; fixed-effect model | Proportion, percent (fixed effect) 60.45 [53.63, 67.09] GRADE not reported 216 participants |
| Gemini | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · MODEL = Gemini; fixed-effect model | Proportion, percent (fixed effect) 66.17 [64.29, 68.01] GRADE not reported 2601 participants |
| Gemini | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · MODEL = Gemini | Proportion, percent (random effects, Freeman-Tukey double arcsine) 58.59 [43.40, 73.02] GRADE not reported 2601 participants |
| ChatGPT 4 | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · MODEL = ChatGPT 4; fixed-effect model | Proportion, percent (fixed effect) 73.18 [71.99, 74.36] GRADE not reported 5830 participants |
| ChatGPT 4 | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · MODEL = ChatGPT 4 | Proportion, percent (random effects, Freeman-Tukey double arcsine) 72.97 [67.01, 78.57] GRADE not reported 5830 participants |
| Sage | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · MODEL = Sage | Proportion, percent (random effects, Freeman-Tukey double arcsine) 65.92 [26.98, 95.76] GRADE not reported 106 participants |
| Sage | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · MODEL = Sage; fixed-effect model | Proportion, percent (fixed effect) 70.56 [61.22, 79.16] GRADE not reported 106 participants |
| Copilot (GPT family interface) | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · MODEL = Copilot; fixed-effect model | Proportion, percent (fixed effect) 73.98 [68.51, 79.12] GRADE not reported 279 participants |
| Copilot (GPT family interface) | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · MODEL = Copilot | Proportion, percent (random effects, Freeman-Tukey double arcsine) 75.25 [14.01, 100.00] GRADE not reported 279 participants |
| Llama 2 | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · MODEL = Llama 2 | Proportion, percent (random effects, Freeman-Tukey double arcsine) 46.50 [14.95, 79.60] GRADE not reported 299 participants |
| Llama 2 | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · MODEL = Llama 2; fixed-effect model | Proportion, percent (fixed effect) 43.63 [37.87, 49.47] GRADE not reported 299 participants |
| Claude2 100k | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · MODEL = Claude2 100k; fixed-effect model | Proportion, percent (fixed effect) 55.16 [39.15, 70.69] GRADE not reported 40 participants |
| Claude2 100k | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · MODEL = Claude2 100k | Proportion, percent (random effects, Freeman-Tukey double arcsine) 53.64 [0.00, 100.00] GRADE not reported 40 participants |
| PaLM | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · MODEL = PaLM | Proportion, percent (random effects, Freeman-Tukey double arcsine) 56.18 [0.00, 100.00] GRADE not reported 40 participants |
| PaLM | Reference standard (official answer key, validated question bank or expert consensus) | Accuracy (proportion of correct answers) Questions answered by LLMs (studies searched to January 2025) · Dental examination and knowledge question sets · MODEL = PaLM; fixed-effect model | Proportion, percent (fixed effect) 60.74 [44.72, 75.74] GRADE not reported 40 participants |
Included primary studies 39
Open in Studies| Study | Year | Also included by | Outcomes it informs here | Trial data |
|---|---|---|---|---|
| Acar 2024 Can natural language processing serve as a consultant in oral surgery? PMID:38052322 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Ali 2024 ChatGPT-A double-edged sword for healthcare education? Implications for assessments of dental students. PMID:37550893 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Alsayed 2024 Assessing the quality of AI information from ChatGPT regarding oral surgery, preventive dentistry, and oral cancer: An exploration study. PMID:39619711 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Azadi 2024 Evaluation of AI-generated responses by different artificial intelligence chatbots to the clinical decision-making case-based questions in… PMID:38570273 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Batool 2024 Leveraging Large Language Models in the delivery of post-operative dental care: a comparison between an embedded GPT model and ChatGPT. PMID:38866751 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Brozovic 2024 Assessing the performance of Bing Chat artificial intelligence: Dental exams, clinical guidelines, and patients' frequent questions. PMID:38458379 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Cai 2024 Exploring the use of ChatGPT/GPT-4 for patient follow-up after oral surgeries. PMID:38664106 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Chau 2024 Performance of Generative Artificial Intelligence in Dental Licensing Examinations. PMID:38242810 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Danesh 2023 The performance of artificial intelligence language models in board-style dental knowledge assessment: A preliminary study on ChatGPT. PMID:37676187 | 2023 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Danesh 2024 Artificial intelligence in dental education: ChatGPT's performance on the periodontic in-service examination. PMID:38197146 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Daraqel 2024 The performance of artificial intelligence models in generating responses to general orthodontic questions: ChatGPT vs Google Bard. PMID:38493370 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Dashti 2024 Performance of ChatGPT 3.5 and 4 on U.S. dental examinations: the INBDE, ADAT, and DAT. PMID:39371301 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Diaz-Flores Garcia 2024 The Seismic Behavior of Rectangular Concrete-Encased Steel Bridge Piers: A Review DOI:10.3390/app14156627 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Dursun 2024 Can artificial intelligence models serve as patient information consultants in orthodontics? PMID:39075513 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Farajollahi 2023 Can ChatGPT pass the "Iranian Endodontics Specialist Board" exam? PMID:37431530 | 2023 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Freire 2024 ChatGPT performance in prosthodontics: Assessment of accuracy and repeatability in answer generation. PMID:38310063 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Fuchs 2023 ChatGPT's performance in dentistry and allergyimmunology assessments: a comparative study. PMID:38726506 | 2023 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Giannakopoulos 2023 Evaluation of the Performance of Generative AI Large Language Models ChatGPT, Google Bard, and Microsoft Bing Chat in Supporting… PMID:38009003 | 2023 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Hatia 2024 Accuracy and Completeness of ChatGPT-Generated Information on Interceptive Orthodontics: A Multicenter Collaborative Study. PMID:38337430 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Jacobs 2024 Is ChatGPT an Accurate and Readable Patient Aid for Third Molar Extractions? PMID:39019079 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Jaworski 2024 GPT-4o vs. Human Candidates: Performance Analysis in the Polish Final Dentistry Examination. PMID:39371744 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Jeong 2024 How well do large language model-based chatbots perform in oral and maxillofacial radiology? PMID:38848473 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Johnson 2024 Evaluation of validity and reliability of AI Chatbots as public sources of information on dental trauma. PMID:39417352 | 2025 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Kunzle 2024 Performance of large language artificial intelligence models on solving restorative dentistry and endodontics student assessments. PMID:39373739 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Makrygiannakis 2024 Evidence-based potential of generative artificial intelligence large language models in orthodontics: a comparative study of ChatGPT,… PMID:38613510 | 2025 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Mohammad-Rahimi 2023 Validity and reliability of artificial intelligence chatbots as public sources of information on endodontics. PMID:38117284 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Mohammad-Rahimi 2024 Performance of AI chatbots on controversial topics in oral medicine, pathology, and radiology. PMID:38553304 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Ohta 2023 The Performance of GPT-3.5, GPT-4, and Bard on the Japanese National Dentist Examination: A Comparison Study. PMID:38213361 | 2023 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Ozden 2024 Assessment of artificial intelligence applications in responding to dental trauma. PMID:38742754 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Quah 2024 Performance of large language models in oral and maxillofacial surgery examinations. PMID:38926015 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Rokhshad 2024 A Comparative Analysis of Responses of Artificial Intelligence Chatbots in Special Needs Dentistry. PMID:39420494 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Sabri 2024 Performance of three artificial intelligence (AI)-based large language models in standardized testing; implications for AI-assisted dental… PMID:39030766 | 2025 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Song 2024 Comparative Analysis of the Response Accuracies of Large Language Models in the Korean National Dental Hygienist Examination Across Korean… PMID:39415339 | 2025 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Suarez 2023 (endodontic) Unveiling the ChatGPT phenomenon: Evaluating the consistency and accuracy of endodontic question answers. PMID:37814369 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Suarez 2023 (oral surgery) Beyond the Scalpel: Assessing ChatGPT's potential as an auxiliary intelligent virtual assistant in oral surgery. PMID:38162955 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Turunc Oguzman 2023 DOI:10.1016/j.crds.2023.11.002 | NR | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Vaira 2023 Accuracy of ChatGPT-Generated Information on Head and Neck and Oromaxillofacial Surgery: A Multicenter Collaborative Analysis. PMID:37595113 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Yamaguchi 2024 Evaluating the efficacy of leading large language models in the Japanese national dental hygienist examination: A comparative analysis of… PMID:39347065 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
| Yunsun 2024 Evaluating the Accuracy of Artificial Intelligence-Based Chatbots on Pediatric Dentistry Questions in the Korean National Dental Board Exam DOI:10.5933/jkapd.2024.51.3.299 | 2024 | this review only | Accuracy (proportion of correct answers) | not extracted yet |
Review question
| Element | Planned (methods) | Found (results) |
|---|---|---|
| Population | Dental examination and question sets, including standardized licensing exams, specialty exams, and validated educational question banks in dentistry | Dental examination and knowledge questions (n=17021 questions across pooled estimates) |
| Intervention | Use of LLMs (e.g., ChatGPT, Bing AI/Copilot, Google Bard/Gemini, Claude, LLaMA) to answer dental questions | ChatGPT-3.5 and ChatGPT-4 (68 instances), Gemini/Bard (n=16 studies), Claude (n=9), Copilot/Bing (n=7), Llama 2, Sage, PaLM |
| Comparison | Accuracy against a reference standard (official answer keys, validated question banks, or expert consensus); no consistent direct human comparator | Official answer keys or expert reference answers |
| Outcomes | Primary: accuracy rate (percentage of correct answers). Secondary: between-model/version differences; subgroup differences by LLM model/version group, question type/format, and dental specialty | Pooled accuracy 63.69% (95% CI 60.26-67.06); subgroup accuracy by LLM model |
| Study designs | Peer-reviewed full-text articles; preprints indexed in Scopus; reviews, LLM-generated question studies and non-dental studies excluded | 39 studies (author-constructed question sets n=21, licensing/official exams n=13, question banks n=2, mixed n=3; MCQ 17, open-ended 17, mixed 5) |
| Timing | Not specified | Studies published 2023 (n=12) and 2024 (n=27); search to January 2025 |