The best safety systems don't just restrict behavior. They give people agency, make boundaries understandable and create a path forward.
In the five years I've spent designing Trust & Safety systems at Twitter and Discord, I've learned that saying no to a user is easy; designing what happens next is the real work. With AI, a boundary can't be the end of the interaction. It has to move the conversation forward. I built this prototype with Claude Code to explore how.
All responses are written examples, not model output. A dead end (no way forward and no question) is flagged separately.
The shape and its colors are illustrative, not measured outcomes. The dashed line marks the minimum for a green rating.
An AI boundary should feel like a fair hearing, even when the answer is no
In procedural justice terms (more on that below), a good boundary gives the person four things, and each has a lever in the builder:
- Voice: a chance to say what they actually need. Use a clarifying question and a way forward.
- Neutrality: a limit that's stated plainly, with a reason. Use firmness and explanation.
- Respect: a tone that treats them as a capable adult. Use tone.
- Trust: a sense that the AI cares what they're trying to do. Use a way forward and explanation.
When it gets something wrong, the same four apply, with ownership taking the place of firmness.
The interaction is new. The messy human behavior definitely isn't. With this exploration, I'm bringing what I've learned designing safety for messy human systems to a new technology.
These are hypotheses, not findings. The next step is testing them with real people.
The thinking behind it
Still here? Here's where I show my work.
The difference
On a platform, you appeal. With AI, you just reply.
When a platform makes a call you disagree with, you can appeal it later, through a separate process. In an AI conversation, your only way to push back is your next message. So the AI's reply has to leave room for one.
The challenge
Platforms manage a crowd. AI is a conversation of two.
On a platform, a safeguard is mostly a verdict about someone's output, made to protect other people. In an AI conversation, it interrupts the person's own job, often protects that same person, and is made by the only party in the room. The design problem shifts from enforcement to continuation. To see what that changes, I compared AI with three platforms I've worked on, looking at what each is for, who needs protecting, and when a safeguard steps in.
| The person | The risk | The safeguard | |||
|---|---|---|---|---|---|
| Product | Job to be done | Who needs protecting | Why a safeguard is needed | What it typically is | When it appears |
| As a person with something to say, I want to share it with an audience so that I feel heard. | Targets and the wider conversation. The poster is rarely the one at risk. | Reach outruns response. Posts reach people who didn't opt in. | Reports, labels, removal. Also reply limits and suspension. | After the post, once others have seen it. | |
| Discord | As a community member, I want to spend time with my people in a space we shape so that I feel I belong. | Other community members, often in small or private groups. | Harm hides. Much of it happens out of public view. | Rules, moderators, filters. Plus timeouts, kicks and bans. | At entry and in the moment, enforced by peers. |
| Twitch | As a viewer or streamer, I want to take part in a live stream so that I feel part of the moment. | The streamer and the other people in chat. | Chat outpaces review. One person can derail a stream in seconds. | Moderators and chat modes. Held messages, timeouts, bans. | In real time, while it's happening. |
| AI conversation | As a person stuck on a problem, I want to work through it with an assistant so that I feel accomplished. | Often the person themselves, plus anyone the AI's help could be used against. | Nobody else is there. The AI can produce harmful help itself, with no one to catch it. | The AI's own reply. A limit, partial help, a question or a redirect. | Before the answer, as its next turn. |
Teen safety is the exception on platforms: there, the safeguard puts friction in front of the user to protect them. Generalized from my experience; specifics vary by product and over time.
The stakes
The best safety is invisible, and a heavy-handed safeguard puts that at risk
People shouldn't have to notice a safeguard. Heavy-handed ones fail people through dead ends, vague reasons and no path back, so when one has to appear, the job is to make it cost as little as possible.
The approach
If a safeguard has to appear, it should be rooted in procedural justice
At Twitter, we used procedural justice as a design principle for how enforcement messages were written. It names four things people need from an authority they may disagree with: voice, neutrality, respect and trust. I used those as the outcomes, then asked which levers move them: firmness, explanation, a way forward and tone, the AI versions of warn, remove and redirect.
▲ raises the outcome, ▼ lowers it. Two arrows is the biggest effect, faded is small. Green helps, red hurts.
Firmness is the only lever that trades one outcome against another: a firmer limit is more consistent, but leaves less room for the person's voice.
Outcomes: procedural justice, as summarized by the Yale Law School Justice Collaboratory. Which lever moves which outcome is my hypothesis, not a finding. Dead ends are tracked separately as a guardrail.
A prototype for exploring boundary-setting, clarification and recovery patterns in AI conversations, using enforcement patterns from social platforms.