A randomized clinical trial in JAMA Network Open tested whether giving physicians access to GPT-4 improves diagnostic reasoning on challenging clinical vignettes. Fifty internists, family physicians, and emergency physicians were randomized to use either conventional tools (UpToDate, Google) or those tools plus GPT-4 for one hour of structured case work.
Result: having GPT-4 on hand did not significantly raise physicians’ diagnostic-reasoning scores—a blinded rubric capturing how well they generated and evaluated differentials, supporting and opposing evidence, and next steps. By contrast, GPT-4 alone, when run with a carefully standardized and pilot-tested prompt, outperformed both groups.
That detail matters: the model excelled under disciplined prompting, but real clinicians weren’t given that scaffolding. The study’s signal isn’t “AI beats doctors,” but that design and interaction quality determine whether large language models truly augment performance.
Scientific Insight
The primary outcome (structured-reflection score, a composite measure of diagnostic reasoning) was similar between groups: median 76% (GPT-4) vs 74% (control), adjusted difference +2 points (95% CI −4 to +8; P =.60). Time per case was also similar (−82 s; 95% CI −195 to +31). GPT-4 alone, using the fixed prompt, scored +16 points higher than control (95% CI 2–30; P =.03). Reliability was strong (weighted κ = 0.66; Cronbach’s α = 0.64).
The authors suggest that prompt quality and minimal user training explain the gap: the LLM performed best when given a carefully engineered, fixed prompt, but clinicians using it ad hoc gained little.
To ensure validity, cases came from a non-public vignette set edited to remove telltale phrases, and mixed-effects models accounted for case and participant clustering.
Leadership Angle
Signal to diagnostics leaders: Access ≠ adoption, adoption ≠ impact.
To lift reasoning quality, tools need workflow-aware design: standardized prompts, built-in reflection scaffolds, and short training, rather than open-ended chat.
The result of “GPT-4 > human doctors” here reflects a lab-grade setup, not real-world autonomy: vignettes lack interviewing, data gathering, and patient context. The next step is prospective workflow trials tied to diagnostic accuracy, testing, and safety outcomes.
Mentorship Angle
For early-career scientists and clinicians, this study itself teaches how to study AI well.
- Measure the reasoning, not just the answer. Chen et al. built a structured-reflection rubric that rewarded how clinicians weighed evidence for and against differentials—a model for rigorous evaluation design.
- Define and pre-register the human–AI handshake. Their fixed zero-shot prompt and blinded grading expose how interface and prompt choices shape performance; future work should test scaffolded vs free chat explicitly.
- Protect external validity. Using non-public vignettes and removing giveaway cues safeguarded against model leakage—a standard every diagnostic-AI study should meet.
This paper shows how careful experimental design lets us see what LLMs can really add to human reasoning (and workflows).