Overcoming the overconfidence problem

When Maximin Lange, Ph.D., and his colleagues at King’s College London and the Massachusetts Institute of Technology began testing large language models (LLMs) to see how well they could decipher medical cases, they began to notice a trend.

“We would watch a model produce a confident, fluent recommendation on a patient whose context it had never been trained on,” Lange says. The recommendation would carry the same numeric confidence score as a textbook case, even though the LLM was operating outside the conditions under which it had been validated.

“A thoughtful clinician in that situation would say, ‘I need more information before I can say anything useful here.’” Lange says. “The model would not.”

And that, he says, was troubling.

In an era where interactions with LLMs have become a common part of everyday life, it has become increasingly easy to assume the confident pronouncements of such models are trustworthy. Yet in an industry like healthcare, built on healthy skepticism and preserving uncertainty, experts like Lange say LLMs will need to develop a new trait before they can safely become clinical advisors: humility.

The benchmark disconnect

Marc D. Succi, M.D., says there is a problematic disconnect between the benchmarks set for LLMs and the real-world reality of clinical care.

“Many [LLM] benchmarks focus on final answers or exam-style questions,” he says. “But real care involves uncertainty, evolving information and stepwise decision-making.”

Succi, who is an associate professor at Harvard Medical School and director of Mass General Brigham’s MESH technology incubator, wanted to see how well LLMs could reason their way through the full workflow of a real-world clinic. To find out, he and colleagues used 29 clinical vignettes to test 21 off-the-shelf LLMs, including multiple iterations of big-name LLMs from OpenAI, Anthropic, DeepSeek, Google DeepMind and xAI. The models were assessed three times across five domains of clinical reasoning.

Succi and colleagues’ findings, reported in JAMA Network Open in April 2026, were curious: The models generally succeeded at diagnosing the “patients,” but their performance suffered in differential diagnosis.

“They were often much stronger at final diagnosis than at maintaining a broad, uncertain differential,” he says. “This is exactly the kind of gap that matters in real clinical care.”

Certainty, not safety

As Succi and colleagues explained in the study, LLMs are trained to recognize patterns, but they “still lack the reasoning processes needed for safe clinical use.”

Clinicians, they explained, preserve uncertainty and carefully refine their differential diagnoses as new information becomes available. LLMs, on the other hand, “collapse prematurely onto single answers.”

Even models that claim to be optimized for reasoning still achieved only incremental improvements over nonreasoning models, and those improvements did not significantly close the differential diagnosis gap, Succi and colleagues found.

Many of the same investigators published a similar study of an earlier iteration of ChatGPT back in 2023. It likewise found the model struggled with differential diagnoses and clinical management.

Succi says the goal of the study was to get ahead of a serious problem before it gets normalized.

“LLMs are already moving toward clinical and patient-facing use,” he says, “but evaluation standards have not kept pace.”

Such distinctions matter because physicians appear eager to use artificial intelligence (AI) in their clinics. A recent American Medical Association (AMA) survey found that 81% of American physicians use AI for tasks such as staying up to date on medical literature and writing discharge notes. Seventy-four percent of respondents said they expect AI to be helpful in diagnosis, and a majority said they expect AI to improve patient outcomes. The only area where a plurality of physicians said they fear AI will be harmful was patient privacy.

Engineering humility

Lange says he’s not surprised that clinicians tend to see risks around privacy and data governance more clearly than the risks associated with relying on AI for clinical guidance.

“When a fluent, articulate system gives you a recommendation that sounds reasonable, the cognitive cost of disagreeing with it is considerable, especially in a busy shift,” he says.

He doesn’t fault physicians; instead, he sees it as a design problem.

“The burden should sit on the system to communicate uncertainty visibly rather than on the clinician to constantly second-guess a tool that sounds certain,” he says.

In a March paper in BMJ Health & Care Informatics, Lange and colleagues unveiled an engineering framework they say can help models leverage uncertainty, just as humans do.

Their framework is called BODHI, which stands for Balanced, Open-minded, Diagnostic, Humble and Inquisitive, and it was designed to help models acquire precisely those traits. By increasing a model’s willingness to seek context and hedge, Lange says their framework can improve the utility of LLM-physician exchanges.

“A humble system changes the interaction,” Lange says. “Instead of producing a declarative answer, it presents reasoning as probabilistic, surfaces the differential, and asks the clinician for the information it needs to narrow things down.”

The framework led one model to increase the rate at which it asked an appropriate clarifying question from less than 8% to 97%. Another model increased its clarifying-question rate from zero to approximately 74%.

Such a system keeps the clinician as the decision-maker, Lange says, rather than turning the clinician into a rubber stamp.

“If clinicians lean on AI to check AI, the field ends up cross-checking hallucinations rather than catching them,” he says. “The only durable anchor is human judgment, and the system must be designed to preserve the conditions under which that judgment can be exercised.”

Pairing with humans

That vision for clinical AI aligns with what industry groups like the AMA have pushed for. In a press release accompanying his group’s survey, John Whyte, M.D., M.P.H., CEO and executive vice president of the AMA, cautioned that “it is critical that augmented intelligence be designed to enhance — not replace — physicians.”

Last year, a team of Pakistani researchers decided to test that premise. Specifically, they wondered whether AI could help fill gaps in low- and middle-income countries, which face higher rates of medical error, partly due to limited resources and greater time constraints. The investigators randomly assigned 58 physicians to review six clinical vignettes using either conventional resources alone or conventional resources plus an LLM. The investigators scored the two cohorts using a diagnostic reasoning rubric. They found that physicians who had the help of an LLM had a mean diagnostic reasoning score of 71.4% versus 42.6% among physicians without access to an LLM. The benefits were particularly pronounced among less-experienced physicians.

Yet, their findings were complicated. In a secondary analysis, the authors found that the LLM alone outperformed the physician/LLM cohort — except when it didn’t. In approximately one-third of cases, the human/LLM cohort beat the solo LLM, suggesting that human judgment often outperforms “knowledge” alone.

A fixable problem

Succi says he believes the issue of overconfidence in LLMs is fixable. He says better training, benchmarks and system design should improve the issue, but he also cautions that one “should not assume that better final answers mean better clinical reasoning.”

Succi says LLMs’ value in the clinic is not so much in arriving at the right final answer but in helping clinicians navigate the uncertainty inherent in practicing medicine.

“Real-world medicine is messy with limited, fragmented data,” he says. “Models need to perform well under these variable scenarios.”

The BODHI framework could be part of such a solution, though Lange cautioned that it is still just a research framework that has been tested only on clinical vignettes. The next step is to see how well it holds up in real-world clinical workflows, working with real-world clinicians.

If future clinical LLMs can learn appropriate humility and find ways to leverage uncertainty the way human doctors do, then they have a chance at being valuable partners, Lange says.

“The goal is not for clinicians to distrust AI,” he says, “but for the AI to earn that trust by being honest about what it does and does not know.”