Distributions, not outputs.

6 min read · pairs with Spread

You have never seen what your prompt does. You have seen one sample of it, once, and decided that was the thing.

What you already know

You would never ship a flow after watching one participant. One session is an anecdote. You run five, or eight, and you look for what repeats — because the thing you care about isn't what happened in that room, it's what tends to happen. The single session is still useful; you just don't mistake it for the finding.

You already have that instinct. You have never pointed it at a prompt.

And nothing has encouraged you to. Every playground on this site — Diff Mode, Tone Dial, Persona Lab — shows you one output per run. That's a fine way to learn a lever, and it quietly trains exactly the habit you'd never accept from a research plan: judging a design by its most recent sample.

What a model actually gives you

A language model doesn't compute an answer. It samples one, from a distribution over many possible answers, and then throws the rest away. Run it again and you get a different draw from the same distribution. Temperature is the width of that distribution — not a weirdness knob, a spread control.

So the honest unit of your work isn't the output. It's the spread. And a clause in your system prompt isn't “working” or “not working” — it holds some percentage of the time.

The lesson

Here is the part that matters, and it is not the part people expect.

A rule that fails every time is not your problem. You notice it on the first run and you fix it. The dangerous clause is the one that fails one run in ten — it passes your spot check, it passes the demo, it passes review, and then it ships and fails for a stranger. Intermittent failures are worse than consistent ones precisely because consistent ones get caught.

Testing once can only ever find the consistent kind. That is the whole argument for running the same prompt ten times: not to be rigorous for its own sake, but because the failures worth finding are invisible at n=1.

It also changes how you read everything else you've built here. Two prompts compared in Diff Mode, one output each, is two coin flips next to each other. Sometimes the difference you see is the prompt. Sometimes it's the draw.

Not everything wobbles equally

Once you start looking at spread, you find the variance is lopsided in a useful way:

• Structure mostly holds. Ask for three bullets and you tend to get three bullets. Format instructions are among the most reliable things you can write.
• Length caps mostly hold, though they drift upward — “one sentence” becomes a long sentence before it becomes two.
• Prohibitions wobble the most. “Never say X” is the classic intermittent clause. It holds until the sentence the model wants to write happens to want X, and then it doesn't.
• Tone wobbles quietly. Not enough to fail a check, enough that a careful reader notices the fourth one sounds like a different writer.

A small example

“No exclamation marks” is about as explicit as an instruction gets. It still only mostly works. That gap — between what you wrote down and what actually happens — is the thing this module is about, and you can only see it by looking more than once.

Turning a feeling into a number

The move is the same one you made in Evaluation: write the criterion down before you look. Take each line of your behavior spec and turn it into something checkable — under 30 words, mentions the product name, no exclamation marks — then score it across every run as a hit rate rather than a verdict.

“Your spec held on 3 of 5 clauses” is a different kind of sentence than “this looks good.” You can act on it, argue with it, and check it again after you edit.

Two honest limits. Checks like these catch mechanical drift — length, forbidden words, required mentions — and say nothing about tone; for that you still have to read the outliers yourself. And a hit rate over ten runs is an estimate, not a measurement: 6 out of 10 means “roughly half the time,” not 60.0%. It's enough to tell a coin flip from a rule, which is all you need it to do.

What to take into the playground

• Run the seeded example first, unchanged. One of the three clauses is written to fail intermittently. Find out which before you read on.
• Now bring your own — paste in a system prompt you already trust from Tone Dial or Persona Lab, and write assertions for the three things you'd swear it always does.
• Expect one of them to be a coin flip. That clause is the finding, and it's the one to rewrite.
• Rewrite it, rerun, and watch the hit rate rather than the output. Going from 6/10 to 9/10 is real progress even though no single run looks different.
• Read the outlier — the run furthest from the median. Assertions can't see tone, and the outlier is where tone drift shows up.