Non-Cochrane · other · PM42153764
The performance of ChatGPT and other large language models on multiple-choice questions in biomedical disciplines: A meta-analysis.
Cheverko Colleen M, Mavrych Volodymyr, Bolgova Olena, et al. · Anatomical sciences education · 2026 · PubMed · doi:10.1002/ase.70262
Findings
25 findings
| Population | ||||
|---|---|---|---|---|
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (all disciplines) | OpenAI GPT-3/3.5 (GPT-3, 3.5, Plus) | None | Accuracy (proportion of MCQs answered correctly) Biomedical education | Pooled proportion 0.593 [0.542, 0.643] GRADE not reported 22 studies · 7627 participants |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (all disciplines) | OpenAI GPT-4 | None | Accuracy (proportion of MCQs answered correctly) Biomedical education | Pooled proportion 0.819 [0.784, 0.850] GRADE not reported 22 studies · 4382 participants |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (all disciplines) | OpenAI GPT-4o through GPT o1 (GPT-4o_o1) | None | Accuracy (proportion of MCQs answered correctly) Biomedical education | Pooled proportion 0.901 [0.880, 0.919] GRADE not reported 8 studies · 996 participants |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (all disciplines) | OpenAI GPT-4 | OpenAI GPT-3/3.5 | Odds of a correct MCQ answer Biomedical education | OR 3.13 GRADE not reported Studies not reported |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (all disciplines) | OpenAI GPT-4o through GPT o1 (GPT-4o_o1) | OpenAI GPT-3/3.5 | Odds of a correct MCQ answer Biomedical education | OR 6.36 GRADE not reported Studies not reported |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (all disciplines) | OpenAI GPT version (GPT-3/3.5, GPT-4, GPT-4o_o1) as meta-regression covariate | None | Variance in accuracy explained by LLM version Biomedical education | Variance explained (meta-regression R2) 0.549 GRADE not reported Studies not reported |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (Anatomical sciences) | OpenAI GPT-3/3.5 | None | Accuracy (proportion of MCQs answered correctly) Biomedical education · Discipline: Anatomical sciences | Pooled proportion 0.632 [0.571, 0.689] GRADE not reported 9 studies · 943 participants |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (Anatomical sciences) | OpenAI GPT-4 through GPT o1 (GPT-4_o1) | None | Accuracy (proportion of MCQs answered correctly) Biomedical education · Discipline: Anatomical sciences | Pooled proportion 0.866 [0.813, 0.906] GRADE not reported 9 studies · 867 participants |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (Biochemistry) | OpenAI GPT-3/3.5 | None | Accuracy (proportion of MCQs answered correctly) Biomedical education · Discipline: Biochemistry | Pooled proportion 0.629 [0.485, 0.753] GRADE not reported 6 studies · 142 participants |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (Biochemistry) | OpenAI GPT-4 through GPT o1 (GPT-4_o1) | None | Accuracy (proportion of MCQs answered correctly) Biomedical education · Discipline: Biochemistry | Pooled proportion 0.851 [0.807, 0.886] GRADE not reported 5 studies · 325 participants |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (Microbiology and immunology) | OpenAI GPT-3/3.5 | None | Accuracy (proportion of MCQs answered correctly) Biomedical education · Discipline: Microbiology and immunology | Pooled proportion 0.500 [0.362, 0.637] GRADE not reported 3 studies · 104 participants |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (Microbiology and immunology) | OpenAI GPT-4 through GPT o1 (GPT-4_o1) | None | Accuracy (proportion of MCQs answered correctly) Biomedical education · Discipline: Microbiology and immunology | Pooled proportion 0.840 [0.783, 0.884] GRADE not reported 4 studies · 223 participants |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (Pathology) | OpenAI GPT-3/3.5 | None | Accuracy (proportion of MCQs answered correctly) Biomedical education · Discipline: Pathology | Pooled proportion 0.648 [0.501, 0.765] GRADE not reported 5 studies · 286 participants |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (Pathology) | OpenAI GPT-4 through GPT o1 (GPT-4_o1) | None | Accuracy (proportion of MCQs answered correctly) Biomedical education · Discipline: Pathology | Pooled proportion 0.746 [0.630, 0.835] GRADE not reported 6 studies · 651 participants |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (Pharmacology) | OpenAI GPT-3/3.5 | None | Accuracy (proportion of MCQs answered correctly) Biomedical education · Discipline: Pharmacology | Pooled proportion 0.613 [0.514, 0.703] GRADE not reported 4 studies · 101 participants |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (Pharmacology) | OpenAI GPT-4 through GPT o1 (GPT-4_o1) | None | Accuracy (proportion of MCQs answered correctly) Biomedical education · Discipline: Pharmacology | Pooled proportion 0.865 [0.811, 0.905] GRADE not reported 4 studies · 227 participants |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (Physiology) | OpenAI GPT-3/3.5 | None | Accuracy (proportion of MCQs answered correctly) Biomedical education · Discipline: Physiology | Pooled proportion 0.645 [0.501, 0.767] GRADE not reported 6 studies · 340 participants |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (Physiology) | OpenAI GPT-4 through GPT o1 (GPT-4_o1) | None | Accuracy (proportion of MCQs answered correctly) Biomedical education · Discipline: Physiology | Pooled proportion 0.830 [0.626, 0.935] GRADE not reported 4 studies · 123 participants |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (USMLE Step 1 and equivalent (not subdivided by discipline)) | OpenAI GPT-3/3.5 | None | Accuracy (proportion of MCQs answered correctly) Biomedical education · Discipline: USMLE Step 1 and equivalent (not subdivided by discipline) | Pooled proportion 0.517 [0.402, 0.630] GRADE not reported 8 studies · 5233 participants |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (USMLE Step 1 and equivalent (not subdivided by discipline)) | OpenAI GPT-4 through GPT o1 (GPT-4_o1) | None | Accuracy (proportion of MCQs answered correctly) Biomedical education · Discipline: USMLE Step 1 and equivalent (not subdivided by discipline) | Pooled proportion 0.878 [0.831, 0.914] GRADE not reported 8 studies · 2360 participants |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (Anatomical sciences) | OpenAI GPT-3/3.5 and GPT-4 through GPT o1 combined | None | Accuracy (proportion of MCQs answered correctly) Biomedical education · Discipline: Anatomical sciences | Pooled proportion 0.77 [0.71, 0.83] GRADE not reported Studies not reported |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (all disciplines) | Anthropic Claude 3 through 3.7 | None | Accuracy (proportion of MCQs answered correctly) Biomedical education | Pooled proportion 0.862 [0.817, 0.897] GRADE not reported 4 studies · 538 participants |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (all disciplines) | DeepSeek R1 | None | Accuracy (proportion of MCQs answered correctly) Biomedical education | Pooled proportion 0.860 [0.780, 0.903] GRADE not reported 3 studies · 345 participants |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (all disciplines) | Google Gemini series | None | Accuracy (proportion of MCQs answered correctly) Biomedical education | Pooled proportion 0.755 [0.680, 0.817] GRADE not reported 10 studies · 1093 participants |
| Single-best-answer text-based MCQs in biomedical disciplines or USMLE Step 1-style assessments (all disciplines) | Microsoft Copilot | None | Accuracy (proportion of MCQs answered correctly) Biomedical education | Pooled proportion 0.694 [0.540, 0.814] GRADE not reported 5 studies · 850 participants |
Included primary studies 37
Open in Studies| Study | Year | Also included by | Outcomes it informs here | Trial data |
|---|---|---|---|---|
| Ch'en 2025 matched by author and year (probable) | NR | 2 other reviews | not stated | not extracted yet |
| Agarwal 2023 matched by author and year (probable) | NR | 1 other review | not stated | not extracted yet |
| Agarwal 2025 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Al-Khater 2025 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Bajčetić 2024 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Bharatha 2024 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Bicknell 2024 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Bolgova 2025a matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Bolgova 2025b matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Chen 2024a matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Chen 2024b matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Duong 2024 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Elkin 2025 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Funk 2024 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Garabet 2024 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Geetha 2024 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Ghosh 2023 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Gilson 2023 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Kaewboonlert 2025 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Keshtkar 2025 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Khan 2025 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Knoedler 2024 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Koga 2023 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Mavrych 2025a matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Mavrych 2025b matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Mavrych 2025c matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Mihalache 2024 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Newton 2025 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Oztermeli 2023 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Ra Vishal 2025 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Sallam 2024 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Salman 2025 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Saowaprut 2025 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Surapaneni 2023 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Tong 2025 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Tordjman 2025 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
| Yaneva 2024 matched by author and year (probable) | NR | this review only | not stated | not extracted yet |
Overlap with other reviews
Reviews that include the same primary studies are not independent evidence. Shared counts match studies by PMID, DOI or NCT, and otherwise by author and year (a probable match). Only the non-Cochrane dental reviews have study lists, so overlap with Cochrane reviews is not counted yet.
Review question
| Element | Planned (methods) | Found (results) |
|---|---|---|
| Population | Single-best-answer MCQs used in biomedical disciplines or USMLE Step 1-style assessments; clinical specialties, veterinary medicine, allied health and dentistry excluded | 41 studies with 207 proportions; anatomical sciences (13 studies), biochemistry (9), microbiology and immunology (5), pathology (9), pharmacology (6), physiology (8), USMLE Step 1/Equivalent (15) |
| Intervention | Empirical testing of LLM performance on biomedical MCQs (zero-shot commercial LLMs without training, priming or fine-tuning) | OpenAI GPT-3/3.5 (k=22), GPT-4 (k=22), GPT-4o through o1 (k=8); Anthropic Claude 3 through 3.7 (k=4); DeepSeek R1 (k=3); Google Gemini (k=10); Microsoft Copilot (k=5) |
| Comparison | Comparison of multiple LLM versions (not mandatory for inclusion) | Newer OpenAI versions versus earlier versions; other companies versus OpenAI GPT-4o through o1 |
| Outcomes | Accuracy: number of correct responses divided by total MCQs administered | Pooled accuracy (proportion of correct MCQ answers); odds ratios from meta-regression of OpenAI version |
| Study designs | Empirical research published in English; MCQs administered in English; non-refereed preprints, retracted articles, reviews, conference abstracts and theses excluded | Cross-sectional LLM benchmarking studies |
| Timing | Studies published January 1, 2022 to August 5, 2025 | Not specified |