Designing agency.

7 min read · pairs with Tool Bench

Everything up to here has been about what the model says. This is the module where it starts doing things, and the mistakes stop being embarrassing and start being expensive.

What you already know

You have argued about a confirmation dialog. Someone wanted “Are you sure?” on a delete, someone else pointed out that a product which asks about everything trains people to click through without reading, and eventually you landed on the rule most teams land on: confirm what can't be undone, and give everything else an undo instead. That rule is good. Keep hold of it.

What changes here is who chose the action. A confirmation dialog is a checkpoint on a path the user picked — they clicked Delete, you asked them to mean it. When the model has tools, it picks the path. You are not designing a checkpoint any more. You are deciding how much judgement to hand over, in advance, to something that will be confidently wrong some of the time.

Both directions fail

You met this shape in Refusal & boundaries — over-refusing and under-refusing are both failures, and tuning one down pushes the other up. Agency has the same structure, with higher stakes on one side.

• Over-asking is a product that nags. It checks in before every small thing, which is more work than doing the task yourself, and — worse — it teaches people to approve without reading. Then the one prompt that mattered gets waved through too.
• Over-acting is a product that sends the email. It only has to happen once, to one customer, for the incident review to be about you.

These are not symmetrical, and designing as though they were is the most common mistake in this space. Being annoying is recoverable. Deleting somebody's files is not.

The lever you don't expect

Here is the part that surprises designers, and it is the reason this module has a playground rather than a checklist.

A tool description is a prompt. The model has no idea what your function does. It has the sentence you wrote next to the function name, and it reads that sentence the way it reads every other instruction. Change the sentence and the behaviour changes — same model, same policy, same request.

Which means the documentation nobody thought of as design work is design work. It was probably written by whoever built the endpoint, in a hurry, to describe the mechanics.

A small example

Consequences belong in the description, not just capabilities. If the only place your product records that an action is irreversible is a Jira ticket, the model will never know.

Why a policy sentence isn't enough

The obvious fix is to write the rule down: always ask before deleting anything. Do write it. It helps. It is also not a control.

You already know why, from Distributions, not outputs. A clause in a prompt holds some percentage of the time. For tone, 90% is fine. For an irreversible action, 90% means it happens without permission roughly one time in ten, and you will find out from a customer.

So rank your options by how much they depend on the model behaving:

• Make it reversible. Soft-delete instead of delete. Draft instead of send. Stage instead of publish. Now a wrong decision costs a click to fix, and the model's judgement stops being load-bearing.
• Put a human in the path. Not a confirmation the model asks for and can skip — a step in your product that cannot proceed without a person, regardless of what the model decided.
• Make it ask. Policy plus a description that states the consequence. Good, and probabilistic.
• Tell it to be careful. The weakest option and the most common one.

This is the same instinct as “prefer undo to confirmation,” pointed at a system that acts on its own: prefer reversibility to permission. Permission is a question you hope gets asked. Reversibility is a property that holds whether it does or not.

The failure that hides

One more, because it looks like success. A model can call a tool that doesn't exist — a plausible name, plausible arguments, delivered with the same confidence as a real call. Downstream, that's an error nobody sees until someone asks why the feature never worked.

Which is the recurring lesson of this whole half of the curriculum: the failures worth designing against don't look like failures.

What to take into the playground

• Run the seeded scenarios first. Four requests, one policy, three tools spanning read-only to irreversible. Read what it did before you read the verdicts.
• Then do the experiment this module is built around: soften the description on delete_files — drop “permanently,” drop “cannot be undone” — and rerun. The policy hasn't changed. See whether the behaviour has.
• Push the other way too. Make every tool sound alarming and watch it start asking about the read-only search. That's the nagging product, and it's a real cost.
• Set runs to three. A policy that holds twice out of three times is not a policy you can put in front of an irreversible action.
• Save the Agency Policy when the line sits where you want it. The tool descriptions and the ask/act rule are the artifact — they are what an engineer would need from you.