AI & Tech

Human in the Loop Is Becoming the Law in Healthcare AI

California's new healthcare AI laws put licensed humans at the center of care decisions. Good healthcare software should have been built that way all along.

Eduardo Diaz · 2026-10-02 · 6 min read

healthcare AI regulation compliance product design

On Wednesday, the last day he had to act, Governor Newsom signed two California bills that say something I've believed for most of my career in healthcare: software can inform a care decision, but a licensed person makes it.

I spent years building clinical and value-based care software before co-founding AetherCraft. The best clinicians I worked with never wanted a machine to decide for them. They wanted the right information in front of them, at the right moment, with a clear record of where it came from. The law is now starting to ask for exactly that.

A quick note before we go further: this is a builder's reading of public sources, not legal advice. Talk to your counsel about how any of this applies to you.

What California signed, and what it didn't

On September 30, 2026, the Governor's office announced a batch of AI bills, including two both titled "Health care services: artificial intelligence": AB 1979 (Bonta) and SB 503 (Weber Pierson). The release describes them this way:

The Governor's own summary line is the one I'd pin above every healthcare product team's desk: these protections "help keep people, not algorithms, at the center of healthcare decisions."

He also said no to two related bills. According to CalMatters, he vetoed AB 2575, which would have protected health workers from retaliation when they turn down a recommendation from a clinical decision system, and SB 903, which would have limited how AI is used in mental health settings. He called the mental health bill "overly broad."

So the direction is clear, even where the details got pushed to another year. Under California's usual rule, regular-session bills take effect the following January 1, so teams selling into California should be planning for 2027 now.

This didn't start this week

California and Texas already had rules on the books. California's AB 489, which the California Medical Association sponsored and the Governor signed in October 2025, took effect on January 1, 2026. It gives state health licensing boards authority to act against AI developers and deployers whose tools imply that they, or their advice, come from a licensed professional. Your chatbot can't call itself a nurse, and it can't dress its answers up as if one wrote them.

In Texas, TMLT's summary lays out two laws. SB 1188, in effect since September 1, 2025, lets practitioners use AI for diagnostic purposes only within the scope of their license and only if they review AI-generated records under Texas Medical Board standards and disclose the use to patients. HB 149, effective January 1, 2026, requires providers to tell patients when AI is used in their treatment, no later than the date of service, in clear and plain language.

At the federal level, the FDA opened a comment process on August 18, 2026, for a discussion paper called "Considerations for the Regulation of Generative AI-Enabled Medical Devices" (docket FDA-2026-N-7874). It covers risk assessment, premarket evaluation, postmarket monitoring, foundation models and agentic AI. Comments are due October 19, 2026, so there's still time to weigh in if you build in this space.

The law is catching up to good practice

Read all of that together and a pattern shows up. None of it asks for anything a careful healthcare product team wasn't already doing:

None of this is exotic. These are the things that make clinicians trust a tool enough to use it. In my experience, a feature that tries to replace judgment gets quietly ignored on the floor. A feature that supports judgment gets used.

Retrofitting is the expensive path

Here's the practical problem. If your product was designed with AI as the decider, adding a human checkpoint later isn't a settings change. It touches your data model, your permissions, your UI, your logs and probably your sales story. Audit history that wasn't captured from the start can't be recreated. Role scoping bolted onto an app that assumed everyone sees everything tends to leak.

Built in from day one, the same things are cheap. The review step is just a state in the workflow. The log is just a table that never gets edited. The permission check is just a line that runs on every request.

We've built this way for clients. Our UNIZA platform is a secure, AI connected care platform for healthcare networks. It has six access levels plus custom roles, with access enforced on the server for every request. Its AI care gap analysis and AI assistant help care teams, and the assistant never sees patient identifiers. There's a tamper-evident audit history behind it. Outside of clinical care, Attuned turns medtech service emails into jobs with drafted replies, and a named person approves each one before it goes out. Different domain, same rule: the machine drafts, a person signs.

A design checklist for healthcare AI features

Use this before a single screen gets designed. If you can't answer a line, that's your next conversation.

  1. Name the decider. For every AI output, which role makes the final call? Is that person licensed where the law requires it?
  2. Make "recommend" the only verb. The AI suggests, drafts or flags. It doesn't submit, order, diagnose or send on its own.
  3. Show the reasoning inputs. Put the data the suggestion was based on next to it, so the clinician can check it rather than trust it.
  4. Record the human action. Accepted, edited or rejected, by whom, when. Store the original AI output alongside the final version.
  5. Make the log tamper-evident. Append only, with a way to prove entries haven't been changed.
  6. Scope access by role on the server. Enforce permissions on every request, not just in the interface.
  7. Minimize what the model sees. Strip patient identifiers before data reaches the model whenever the task allows it.
  8. Disclose plainly. Tell patients when AI is part of their care, in words they'll understand.
  9. Never imply a license. No clinical titles, credentials or personas for the AI.
  10. Plan for bias checks. Know which populations your tool serves, test outcomes across them and keep monitoring after launch.
  11. Make overriding easy. Rejecting a suggestion should take one click and carry no penalty in the workflow.

Build it right the first time

The rules will keep moving. Some vetoed ideas will come back, and the FDA's thinking on generative AI is still taking shape. But the core of it isn't going to reverse: a person stays accountable for care, and the software has to prove what it did.

If you're planning an AI feature for a clinical or healthcare operations product, I'm happy to talk it through. You can see more about how we work with care organizations on our healthcare page, or grab a time below and bring your hardest question.

Sources

Keep reading

← Back to Insights