Judging at scale.

7 min read · pairs with Judge Lab

Scoring by hand stops working somewhere around the fiftieth output. Handing it to a model is the right instinct. Trusting what comes back is not.

What you already know

You have counterbalanced a study. Two prototypes, and you did not show them to every participant in the same order — half saw A first, half saw B — because you know perfectly well that whichever comes first has an advantage that has nothing to do with the design.

Nobody taught you that as a special technique for AI. It is just what it means to run a comparison properly. An uncounterbalanced preference test isn't weak evidence, it's not evidence.

A model asked to compare two answers is a participant. You have never counterbalanced it.

Why automate at all

Because the alternative doesn't scale, and pretending otherwise helps nobody. A rubric applied by hand across a dozen cases is a good afternoon's work. Across a thousand, on every release, it doesn't happen — which in practice means nothing gets evaluated at all.

So automate. The mistake isn't automating; it's treating the output as a measurement before you have checked that the instrument measures anything.

The check

Run every comparison twice, with the two answers swapped.

A judge reading quality picks the same answer both times. A judge reading position also gives you a consistent-looking result — it picks the same slot both times, which is a different answer. One swap tells those two apart, and nothing else does.

Why agreement is the wrong headline

This is the part worth slowing down for, because the number everyone reports is the one that means least.

Say your judge agrees with your own picks on eight of ten pairs. Sounds like a working instrument. Now consider a judge that has learned nothing at all and simply always names the first answer. With two options it will be right about half the time by construction — and if your examples happen to list the better answer first more often than not, its agreement climbs, with no judgement involved anywhere.

Agreement without consistency isn't a measurement. It's the rate at which two things coincided. Which is why the flip rate goes first, and the agreement number gets reported only over the pairs that held steady.

A small example

Neither reply looks like a failure. Both give a rationale. Read either one on its own and you would take it seriously — which is why the check has to be mechanical rather than a matter of noticing.

The other things that sway a judge

• Length. Verbosity reads as thoroughness. A longer answer restates the question, adds empathy, hedges — and scores higher for it, on criteria that never mentioned length.
• Position. The one above, and the cheapest to test, which is why it's the one the playground automates.
• Self-preference. A model asked to grade its own output has a stake in the answer. Worth knowing about even where it's awkward to test.
• Vague criteria. The root cause of most of the above. Told to pick the “better” answer with nothing operational to go on, a judge falls back on whatever surface features are available — and length and position are the most available things there are.

That last one is the actionable one. A high flip rate is usually a problem with your criteria before it is a problem with the model. You wrote a rubric in Evaluation precisely so “good” would stop being a feeling; a judge handed a vague rubric has the same trouble a new reviewer would.

The awkward recursion

You are using a model to check a model's work, and the checker has the same failure modes as the thing being checked. There is no way around that by adding another model on top.

What there is a way around: making the check something you compute rather than something you trust. Swapping two answers and comparing the verdicts isn't a judgement about the judge — it's arithmetic. That's the whole reason it works.

What you can actually do

• Counterbalance by default. Every comparison, both orders, every time. It doubles the cost and it is the difference between a number and a rumour.
• Report the flip rate next to the agreement rate. An agreement figure quoted on its own should get the same reception as a usability finding from one participant.
• Fix the criteria before blaming the model. If the judge is flipping, read your own rubric and ask what it actually told anyone to look for.
• Use the judge to triage, not to decide. It is genuinely good at finding the fifty outputs worth your attention out of a thousand. That is a different job from signing off on a release.

What to take into the playground

• Make your own picks before you run anything. A call made after seeing the machine's answer isn't ground truth, it's agreement.
• Run the seeded set. In all three pairs the shorter answer is the better one, so a length-biased judge and a position-biased judge fail in visibly different ways.
• Read both rationales on any pair that flipped. The same model arguing confidently for opposite answers, minutes apart, is more persuasive than any argument this article can make.
• Then sharpen the criteria — replace anything that isn't checkable with something that is — and rerun. Watch the flip rate, not the agreement.
• Save the Calibrated Judge once the flips are gone. The criteria plus the calibration result are the artifact: they are what lets someone else trust a number this judge produces.