Disclaimer

On noble and imperfect AI policy

This week, my home state of Washington passed a bill adding mental health safeguards to “companion chatbots.” It requires AI interfaces to notify users at the start of every conversation, and again every three hours, that they’re talking to an AI (versus a human). For minors, the reminder comes every hour. It’s expected to go into effect January 1, 2027. There are more provisions around detecting self-harm and suicidal ideation, prohibitions on manipulative engagement techniques, and civil liability under the Consumer Protection Act. I’ll leave those to people with the relevant expertise, and focus on the notification part. For now, let’s agree that attempts at AI policymaking are a positive thing, and also sometimes reveal a misunderstanding of how these products are actually built.

The intent is good. Wrongful death lawsuits have been filed against AI companies, kids have been harmed, and people in vulnerable states have formed deep attachments to companion products that were designed, quite deliberately, to foster exactly that kind of dependency. The sponsor of the bill, Rep. Lisa Callan, compared companion chatbots to a predator, which isn’t a soft read of the situation. A growing number of states are trying to get something on the books. The instinct to intervene is admirable, but the mechanism of a simple disclaimer sent over and over points to a larger problem with how AI gets regulated (or doesn’t).

Two Systems

The long and short of it, which often gets lost in the hullabaloo, is that the model and the interface are not the same thing, and they don’t automatically know what the other is doing. This has always been true. When you open a companion app and start a conversation, you’re interacting with a front end, a UI that a product team designed, with whatever text and buttons and copy they decided to put on the screen. Underneath that is the model, which is a separate system entirely, trained on data, shaped through a long and expensive process, and then deployed into that interface.

Oddly, though reasonably, the model can’t see its own screen. It doesn’t know “at birth” what the app is called where it lives, what the terms of service say, what the onboarding flow looked like, whether there’s a banner at the top of the window, or what disclaimer text might be appearing every three hours above the chat. It knows what’s in the conversation at hand, and you can tell it explicitly about itself through training and configuration (which is how you get your Claudes, Groks, etc. knowing who they are and who built them), and that’s more or less the extent of it.

That means a law requiring a disclaimer to appear in an AI product’s interface every few hours hasn’t changed anything about how the model behaves. The model underneath is still operating on its own logic, shaped by its own training, and it has no idea the disclaimer is there. It will not factor it in. The conversation continues exactly as it would have without the law (again, this is probably where the other levers in the bill will come into play more strongly, and I’m not arguing that they won’t; only how I see this particular provision playing out).

Getting a model and an interface to work together coherently is hard work that happens imperfectly even inside the companies that do it every day. It requires building that coordination into the model’s training, into its system prompt, into the instructions it carries into every conversation. If you want the AI itself to understand something about its own product context, like what the interface has told the user, or what commitments have been made on the screen, you have to put that in the guts, too. A designer adding a text box to the front end has (probably) not communicated anything to the system generating the responses. And this can cause a real experience gap even in innocuous cases, let alone an acute mental health crisis.

Layered

So when a bill creates a protective intervention by adding disclosure text to a user interface, it’s working on the wrong layer of the problem. The layer that surfaces to users, that can be more easily regulated through legalese and product requirements, is not necessarily the layer where the behavior happens that causes harm. Whether a model knows to slow down when a conversation is heading somewhere dangerous, or understands how to redirect rather than validate, relies on mechanisms below the user interface. You can even engineer canned responses that override the model, and are still not the model, even when that language appears inside the conversation thread (this is how some helplines are presented). You can mandate the wrapper all day without touching what’s inside it.

Which means compliance becomes a little too easy. A company can add the required text to the screen, check the regulatory box, and ship a model that was trained without any of this in mind. The notification appears every three hours, a user in a vulnerable state reads it, or doesn’t, and the conversation resumes because the system generating that conversation was never told anything about it.

Different AI companies are all over the place with how much they care. So I suppose this bill is trying to be a catch-all, which would target those who haven’t done the work to clear the bar alongside more responsible actors. That’s alright. But again, the work to train guardrails and to catch certain patterns before they compound happens at a level that’s largely invisible to users and lawmakers, which causes all kinds of misunderstandings and gaps between policy and lived experience.

Dig Deep

What might actually reach the problem is some external standard for how these systems are supposed to handle users who are clearly not okay. Right now, it is literally not your business what an AI company is doing day to day. Maybe they’re asking the right questions, maybe they’re red-teaming, maybe they’re definitely not. What does the model do when someone has been in a conversation for six hours? What does it do when the emotional register of the conversation has been escalating for days? Does it have any concept of when it’s caught in a sycophantic loop? The answers are calibrated to whatever internal threshold the team decided on.

AI regulation is HARD. It’s hard to write, it’s hard to enforce, it’s hard to keep up. It requires technical fluency in the regulatory process that doesn’t really exist yet in any consistent way. It also requires getting into territory that AI companies guard carefully, because the model is the product, and it’s easier to insist on legal disclaimers than full overhauls of intellectual property. It would mean regulators sitting across the table from ML engineers and having the conversation at that level, which I don’t think is a popular choice for either side.

I understand why that reach hasn’t happened. Everyone has something to protect. I’m not certain what all this should look like, and I’m not sure anyone is yet. What I do think is that a policy premised on the the interface rather than the system reads more like compliance than true protection. January 2027 is a long time away in AI development time. The products that will exist when this law takes effect will be more capable, more voice-forward, and more deeply woven into people’s lives than ever. Everyone needs to dig a little deeper before then to understand what that means.