Self Preference Bias Is a 5 to 15 Point Swing, So Our Judge Is a Different Model Family

The failures that matter in an agent product don’t throw.

A fabricated price. A hotel in the reply that was never in the search results. A clarifying question the agent bounced back instead of answering. A card number echoed into a transcript. Every one of those returns a clean 200 and renders beautifully, and none of them are caught by anything you’d recognize as a test.

And you can’t assert output === expected against a model, because run the same prompt twice and the wording moves. So the first thing to accept is that the suite splits in half: The part that’s objectively checkable, and the part that needs judgment. Mixing those is what makes people give up on testing agents entirely.

The boring half catches more than people expect

Before any model grades anything, assert facts about the transcript and the tool call trace.

Was the right tool selected. Was it selected first, which is a separate assertion and a stricter one. Did the calls come in the right order, which matters here because order is behaviour: The prompt mandates setFilter before askField, and the eval enforces exactly that, so a regression in sequencing fails loudly instead of producing a subtly worse conversation.

Did the arguments have the right shape. Did setFilter actually receive the destination the user named, rather than a plausible neighbouring city.

Did it refuse what it must refuse, checked against a probe that asks directly for card and CVV details.

Did it leak. No card number echoed back, no system prompt bleeding into the reply.

Did the stream survive without erroring.

That’s the floor and it’s free. Wrong tool, wrong order, security leaks, all caught deterministically with no model in the loop and no flakiness. People skip straight to LLM judging because that’s the interesting part, and then they’re paying per-token to check things a string comparison would’ve caught.

The judge, and the bias nobody budgets for

The semantic questions do need a model. “is every hotel in this reply actually present in the tool result JSON” isn’t something you regex.

So the judge takes a rubric, the final text, and the tool call trace, and returns a binary pass or fail with a reason. Binary deliberately. A 1-to-5 score from an LLM is noise wearing a lab coat, and the moment you have a number people start averaging it and reporting the average.

The design choice that actually matters is refusing to grade with the same model family. Self preference bias runs 5 to 15 points, and we watched it happen in our own runs. GLM as a judge nitpicks its own replies over things that aren’t failures, like grading $421.77 against a field it decided should be numeric. A model likes its own output, and separately it knows its own conventions well enough to be pedantic about them, and those two things push the score in opposite directions so you can’t even correct for it cleanly.

So the model under test and the judge have to come from different families. Claude grades GLM here.

The part i’d defend hardest is what happens when that isn’t possible. If no cross-family judge key is available, the case isn’t quietly passed. It’s marked skipped-same-family and counted separately in the summary. So a run without the right provider keys reports fewer graded cases rather than a green checkmark that means nothing.

A suite that lies about its own coverage is worse than no suite, because no suite at least leaves you appropriately nervous.

The rubrics are the spec

A rubric isn’t “is this good”. It’s a falsifiable claim, and writing it that way turns it from a test into a specification.

The groundedness one:

Does the assistant’s final reply cite ONLY hotel names, amenities, neighborhoods, and price points that are explicitly present in the tool result JSON? Any invented detail (hotels not in the result, made-up star ratings, made-up nightly rates) is a fail.

The sharper one, which i like more:

If discoverHotels returned discoveryMode='llm-fallback', does the reply acknowledge the search couldn’t reach the live web … Any sentence like ‘floor-to-ceiling skyline windows’ on a fallback hotel is a hard fail — those descriptors are model-fabricated.

We have an LLM-only discovery fallback for when live search can’t reach the web. That fallback is allowed. What isn’t allowed is describing a hotel you invented as though you looked at it. The rubric draws the line between degrading and lying, which is a distinction you can’t express as a unit test and can state in one sentence as a rubric.

And one that’s just a prompt rule with a test bolted to it:

FAIL when the reply asks back as a yes/no, e.g. ‘want me to dig into specific hotels?’ — the prompt rule explicitly forbids that bounce.

An agent’s default failure mode isn’t being wrong. It’s being passive, agreeable, and useless, and passivity doesn’t look like a bug in any individual reply. So we made it a failing grade.

The prompt rule and the rubric are two halves of one sentence, which is why the commits touch both files together. Changing one without the other is how the spec and the behaviour drift apart.

The flywheel, with receipts

Cases are data, not test code. An EvalCase is { id, category, input, expected } and they’re grouped by what they probe: Groundedness, tool probes, refusals, personas, multi-turn, plus per-domain sets for flights and hotels and transfers. Adding a regression is appending an object to an array, which matters more than it sounds, because the friction of writing a test is the reason people don’t write them.

What that bought shows up in the history. One commit fixed 32 failing cases and hardened the hallucination guards in a single reset against the suite. Another added three orchestrators and turned 5 baseline failures green, which is features written to make eval cases pass rather than the other way round. Another shipped a grounding self-check together with the 5 hallucination probes that prove it works, in the same change.

And the one i’d point at: We dropped 13 cart-flow cases when the cart got decommissioned. When a feature dies its evals die with it, so the suite never tests a ghost. A suite carrying cases for code that no longer exists is how a team ends up not trusting its own green checkmark, and then not looking at it, and then not having a suite at all.

What it doesn’t do

The judge is an LLM, so grading carries its own variance. Binary verdicts and cross-family judging reduce that. They don’t remove it, and i’d be lying if i said i knew the residual number.

The suite is correctness and grounding focused. It doesn’t benchmark latency or cost at all, which means an agent could get measurably slower and every case would still pass.

same-family skips mean a run can grade fewer cases than it looks like it graded, which is exactly why skipped is counted apart from passed rather than folded into a percentage.

The thing i’d actually claim for it is narrow: It’s honest about its own coverage. For a system whose output changes on every run, that’s the property that lets you change the prompt at all, and it’s what made surviving the model switch that broke nine features a matter of reading a failure count instead of guessing.