AI’s allure drives some users into the shadows

Asha Palmer, J.D., has an analogy for shadow AI.

“It’s like the grim reaper.”

Palmer, the senior vice president of compliance solutions at the technology firm Skillsoft, makes the comparison in jest, but not because the threat is anything short of serious.

As artificial intelligence (AI) becomes ubiquitous across all sectors of business and daily life, individual health systems and clinics have taken a wide range of approaches to AI integration. Some are cautious, limiting its use to only the most basic tasks, like helping people find information on the health system’s website. Others have become enthusiastic early adopters, piloting AI approaches to high-stakes use cases, such as clinical decision support.

None of that, though, is what Palmer means when she invokes the shadowy figure with the scythe. There may be pros and cons to such uses of AI, but at least those institutional uses are based on some sort of vetting process; their use is out in the open.

The term “shadow AI” refers to all of the ways employees might be using AI outside of the umbrella of sanctioned uses. Perhaps an employee uses ChatGPT to help draft discharge notes, even if ChatGPT is not approved by the health system for such uses. Staff might install a browser extension to summarize websites, even though the extension has not been evaluated by the organization’s information technology (IT) department. Or a physician could use AI to help analyze patient data, without clear guidelines about how to crunch data in a way that does not violate patient privacy regulations.

None of this is unique to healthcare, Palmer says, but when the data at stake are health data, the stakes are incredibly high.

“In compliance, we have the general rule when we educate employees: Don’t give these tools anything that’s not yours, right?” she says. “Because you can’t get it back.”

No thinking about it

Although many AI products are tailored to meet the needs and regulations of the healthcare industry, the vast majority of AI applications are focused on improving efficiency, and many of those are either free or have a free version. In other words, the barrier to usage is low.

All of this matters, Palmer says, because health companies need to know how the data they input are being used. “[The] question is, where does that information go?” she says. “How is it used, and what is it used for? And is that actually positive or negative?”

Palmer says she suspects most healthcare companies are aware that employees are likely using shadow AI, but she fears that knowledge does not always lead to action.

“I do think most people are choosing not to think about it, because the reality is thinking about it makes it a lot harder to govern and a lot harder to enforce the governance structures around it,” she says.

Healthcare organizations might hope the problem stays contained or does not lead to major negative consequences. Palmer says a much better strategy is to be proactive. After all, she says, “hope is not a strategy.”

A “mainstream behavior”

The available data suggest that shadow AI use has already become a major problem. In December, Black Book Market Research published the results of a survey of 228 U.S. health system employees. The respondents included 92 enterprise and service-line leaders and 136 frontline healthcare workers.

The main takeaway was that health systems are eager to launch AI pilots to test out potential new efficiencies. But their survey also showed troubling data points related to shadow AI. It found 58% of frontline respondents said they use generic AI tools like OpenAI’s ChatGPT, Google’s Gemini or Microsoft’s Copilot to do work-related tasks at least once a month, and 39% of frontline workers who filled out the survey said they use it at least once per week.

Of those generic AI users, 17% said they “sometimes” or “often” include identifiable patient information when they use AI. Another 27% said it was “rare” for them to use private patient information, but they conceded that they occasionally “slip up.”

Douglas Brown, M.S., Black Book’s president, said the data make the issue crystal clear.

“Shadow AI isn’t a fringe problem,” he said in a news release. “It’s a mainstream behavior emerging in environments where official tools don’t keep up with real-world needs, and where policy is vague or absent.”

Finding solutions

Samantha Tirado, director of policy communications for the Medical Group Management Association (MGMA), told Managed Healthcare Executive that medical groups are working hard to better understand how to safely implement and use AI tools.

“We advocate for policies that recognize [that] responsible AI adoption involves governance,” she said.

She said MGMA’s work on this front has involved advocacy at both the practice and policy levels. Practices need tools and resources to help them understand how best to use AI, and MGMA has made available a number of resources to its members on that front. She said her organization has also been urging regulators to establish clear guidelines that allow medical practices to use the tools with confidence without running afoul of privacy and other regulations.

Tirado noted that the HHS has proposed changes to the Office of the National Coordinator for Health IT Certification Program. If approved, those changes would eliminate some of the requirements that healthcare technology firms explain how predictive or generative AI was designed, how it makes decisions and how it uses data. The agency says such regulations may not be necessary given that other federal health privacy rules already apply.

However, in a letter to Thomas Keane, M.D., MBA, the assistant HHS secretary for technology policy, Anders M. Gilberg, M.G.A., senior vice president of government affairs of MGMA, argued that getting rid of the standardized transparency rules could complicate governance of AI.

“The potential loss of standardized transparency could make it more difficult for practices, especially smaller and community-based medical groups, to evaluate AI tools for adoption, support informed provider use, and train staff appropriately,” he wrote.

Gilberg added that such a change would put medical practices in the position of having to choose between relying on assertions made by technology companies and performing their own independent assessments of such technology.

“This creates a gap at a time when guardrails, governance, and trust in AI-enabled decision support are increasingly important,” he wrote.

Out of the shadows

Palmer says one of the problems with shadow AI tools, particularly when they are free, is that the AI developers do not have any legal responsibility to be transparent to their nonpaying customers.

“The reality is, unless you’re buying an enterprise license for these tools, they’re not [transparent],” she says. “Their standards will not satisfy a healthcare organization standard. I mean, that should be stipulated.”

She says enterprise clients of AI companies, though, should have the ability to get transparent answers from their vendors, which is necessary to successfully implement AI.

Although tackling the problem of shadow AI may seem daunting, Palmer says it starts with very practical solutions. Healthcare leaders should start by having an honest discussion about which AI tools they are using, which they want to use and for which purposes they are comfortable using AI. MGMA recommends setting up clear guidelines about data usage, when human review of AI is necessary and what that review should entail. They also recommend setting up processes for employees to report misuse of AI, potential bias in AI outputs and potential AI hallucinations.

What organizations should not do, Palmer says, is create a culture in which employees feel like they have to hide their AI use.

“What we know is that once people start, there’s a slippery slope,” she says. They may start to use it for smaller tasks but then begin to trust the AI system more and more until they are suddenly using it for more complex, potentially risky tasks.

In one sense, Palmer says, the danger of shadow AI is less about the technology and more about its ability to make people forget their highest calling — caring for patients.

“Because the reality is, these software products are extensions of our care, right?”

Jared Kaltwasser is a medical writer in Iowa and a regular contributor to Managed Healthcare Executive.