Agentic AI may take humans out of the loop

James F. Hennessy, J.D., works in San Francisco, so he’s gotten used to seeing driverless taxis escorting people around the city. But as a lawyer, he also has some questions. For instance, what if a police officer sees a robotaxi speeding? Whose name would go on the speeding ticket?

“Would they cite the vehicle? Would they cite the software developer of the vehicle? Would they cite the passenger?” he says. “That would be kind of weird.”

It might seem trivial, but Hennessy, a partner in Reed Smith’s Life Sciences Health Industry Group, says autonomous vehicles are an important illustration of bigger questions coming down the road. Just because human drivers cease to be necessary does not mean the rules of the road—speed limits, stop signs, and parking regulations—cease to exist.

“Right now, we’re in that awkward phase where now we have autonomous robots roaming around on our streets, and we are trying to apply old rules to new technologies, and it’s awkward,” he says. “And that’s kind of where we’re at in healthcare.”

The early days of artificial intelligence (AI) in healthcare focused on AI’s ability to perform particular skills, such as summarizing information or crunching data to predict healthcare outcomes. Yet, the emerging wave of AI products goes beyond performing discrete tasks that people ask it to do. Agentic AI can act autonomously to achieve a given outcome, updating and refining its approach without human intervention. It’s like the difference between a [global positioning system] providing possible routes and a robotaxi driving the passenger to the destination.

In theory, agentic AI’s ability to self-refine could mean improved patient outcomes. Hennessy says it also raises questions for regulators and end users, because assessments of AI agents will quickly become outdated as those agents evolve. An agent that is approved by the FDA in January might operate substantially differently by July.

Hennessy said the FDA has been trying to adapt to keep up with the evolution of healthcare technology.

“But none of [those changes] are really designed to have certain technology that can just evolve and be working independently toward a goal, rather than something that has a static algorithm,” he says. “It’s a completely different world.”

Evolution of liability

The issue of agentic AI is particularly fraught when AI is used in the service of clinical care. As humans begin to rely on such technologies, they will face questions of safety and liability.

recounted a conversation with a colleague who described the pros and cons of using an algorithm designed to monitor patients for sepsis. The algorithm can do a good job in many cases, but the colleague noted that an AI system is limited by incomplete information. She described a scenario in which an algorithm might advise pushing fluids and giving an antibiotic to a patient.

“The agent, or the AI algorithm, does not know that those directions will kill this patient because they have kidney failure,” says Snowdon.

With a static AI system, a user might identify such blind spots and account for them. However, an ever-evolving AI agent might develop blind spots that the user is unaware of. If a clinician were to follow the agent’s recommendations and harm the patient, it could raise high-stakes questions about who is liable: the AI agent or the human who consulted it.

Hennessy says there is not yet sufficient case law to determine exactly how such problems will be adjudicated. However, he says that physicians are licensed professionals and thus remain accountable to state licensing boards.

“The rules still require that professional medical services be furnished by licensed healthcare professionals,” he says. “So one question will be how state medical boards deal with this.”

‘Orchestrator’ in the loop

Basile Njei, M.D., M.P.H., Ph.D., MBA, an associate professor adjunct at the Yale School of Medicine who has studied AI extensively, says agentic AI will force a reconceptualization of the role of humans. AI agents may improve themselves over time, but that does not excuse humans from playing a critical role in patient care. He says it’s not enough for humans to be “in the loop.” They need to be in charge.

“The human being is going to be the orchestrator,” he says.

For Njei, the term “orchestrator” better captures the idea of the human being as the ultimate agent. He says humans should look at the work of agentic AI in the same way they look at human work. It could produce good, even expert-level work. It could get better at its job over time. Ultimately, though, a human orchestrator needs to take responsibility for understanding the work of his team — whether human or AI work — and then making an informed decision.

Limits of explainability

Still, Njei says being the orchestrator is getting more complicated. Historically, a key element of decision-making has been understanding the thought process of the people making recommendations. That gets more difficult when a recommendation comes from an AI system, and the difficulty increases further when the processes it uses evolve autonomously over time.

Njei says human orchestrators need to adapt their approach because the concept of explainability is intertwined with certain logical frameworks that themselves might be flawed.

“You’re going by your current standard of care or standard of thinking, and some of the things we thought were standard of care 10 years ago are not standard anymore, because we were wrong,” he explains.

Njei says clinicians need to leave room for potential solutions that are not yet easily explainable. After all, part of the allure of AI is its ability to spot patterns and develop potential solutions that humans have not yet thought up. That’s why he says clinicians should be open to solutions that are reproducible even if they are not yet explainable.

“Because if something keeps repeating itself in different data sets, different populations, different subgroups,” he says, “it doesn’t really need to be explainable to be true.”

Life-cycle management

Snowdon says one part of the solution to the problem of agentic AI is having a robust life-cycle management program in place to assess such tools continually over time.

“What is the purpose of this particular AI technology, and how is it advancing an organization’s strategy?” she says. “And the critical question is, what’s the oversight? Whose role is it? Who is accountable?”

She says organizations need to be clear about which humans will be in charge of ensuring that technologies actually achieve what they were meant to achieve, and that these goals are achieved even as the agent iterates. She notes the AI isn’t the only thing that can change.

“Models drift, but so does your data shift,” she says. “People showing up in the emergency room post-COVID-19 are not the same patients they were pre-COVID-19.”

Njei says it will also be important to test AI products with the same level of rigor as we test other healthcare products. Specifically, he says, such products need to be tested in the clinic, not just using simulations. They also need to be assessed on their ability to pass a trustworthiness test for patients and clinicians.

He says regulators and users should also deploy so-called verifier agents, independent agentic AI systems that can continually validate the models being used in clinics. All of this, though, will not happen overnight, he cautions.

“I think that we will need to retrain almost everybody,” he says, “and constantly train them every six months…and most people are not doing that.”

Jared Kaltwasser is a medical writer in Iowa and a regular contributor to Managed Healthcare Executive.