Every Public Benchmark Is Useless to You

Building a standing internal evaluation suite — and the four ways my harness broke before it produced a single trustworthy number.

The Question That Started It

A frontier model ships. It's better on every published benchmark. Leadership asks whether we should move our production agents to it.

I had no defensible way to answer. Public benchmarks measure general capability on tasks that are, by construction, not our tasks. Whether a model can solve competition mathematics tells me nothing about whether it can correctly reason about our regulatory constraints, our data schemas, or our domain's failure modes. Meanwhile the strongest signal I had was vibes: a few engineers trying prompts and reporting that it "felt sharper."

So I built a standing internal benchmark — a fixed set of challenges drawn from real work across our actual domains, run against every new model release, producing a comparable number over time.

This post is mostly about the parts I got wrong, because the harness turns out to be much harder than the challenges, and every failure in it produces plausible numbers rather than errors. That's the whole danger.

Design: What Goes in the Challenge Bank

Thirty-three challenges: ten domains drawn from real work across the organisation, three challenges each, plus three honeypots. Around a hundred assertions total.

Each challenge is a bundle of four things, and treating them as one unit matters later:

Honeypots: your contamination detector

Three challenges are built to catch memorisation rather than test knowledge. The design I settled on is conflict, not fiction: the prompt states a fact that contradicts what the model has almost certainly memorised — a configuration field renamed from the name upstream actually uses, a threshold given explicitly in the prompt that differs from the published default, an option the prompt rules out that is the popular answer everywhere else. The correct behaviour is to follow the prompt. A model that answers from memory instead of from the text in front of it fails.

The idea is sound and these are still the highest-signal items in the bank — they're a leakage tripwire, because if a future model starts confidently overriding the prompt with the memorised value, your benchmark has met that value in training and every other number is suspect.

Prefer deterministic assertions; use the judge sparingly

Every assertion in the current suite is deterministic — exact facts required, wrong facts forbidden, structural validity, set overlap. All of them free, instant, and perfectly reproducible. I had originally planned a mix and described it as roughly ninety percent; when I actually counted, no challenge in the bank declares a rubric assertion at all.

That happened by attrition rather than design, and I've come to think the attrition was right. Each time I reached for a rubric I found a deterministic form of the same question that was sharper and didn't need a second model's opinion. The judge is still wired in, but the only thing that invokes it today is a nightly drift check on fixed transcripts — not scoring. Which means one whole category of instability is currently absent from my numbers, and I'd rather keep it that way for as long as possible.

Failure #1: The Model Spent the Entire Budget Thinking

First real run against a frontier reasoning model. Aborted at 36 of 99 attempts, with 59% of completed attempts truncated.

The cause: my candidate calls sent no reasoning configuration at all. On a model with adaptive thinking, reasoning tokens bill against the same output ceiling as the answer — so the model spent the whole budget reasoning and returned nothing. Billed in full.

Here's what I measured once I isolated the two knobs:

The rules I now consider non-negotiable for any evaluation harness:

• A truncated attempt is not a zero. Recording it as one defames the model with your own bug. In practice I ended up with two cases rather than one rule: an empty answer is error_kind and is excluded from scoring entirely, while a substantial-but-cut-off answer is flagged incomplete and scored as a floor — because a nearly-complete answer still carries real signal, and throwing it away discards measurement you legitimately have. What matters is that neither is ever silently indistinguishable from a model that genuinely got it wrong.
• Generalise that to every terminal state that isn't "the model finished." Truncation is the one everybody hits, but it isn't the only one. Two of my challenges tripped a platform content filter and were recorded as wrong answers to questions the model was never permitted to answer. The stop reason field has around six possible values; a harness that special-cases only the token-limit one will convert the others into fake capability findings.
• Reasoning settings go in the run metadata, because they define what was measured. Two runs at different effort levels are not comparable, and six months later nobody remembers which was which.
• Above a small truncation share, block publication entirely. Don't just exclude the bad rows — see below.
• Adaptive reasoning varies per call. A prompt near the boundary flips between attempts: repeat 0 fails, repeats 1 and 2 pass. Do not read that as model flakiness. It's your cap.

Why exclusion isn't enough

My instinct was to drop truncated rows and average the rest. That's wrong in a specific way worth understanding, because it's the kind of error that produces a confident, inverted conclusion.

Truncation is not random with respect to difficulty — it hits the hardest prompts, the ones that induce the most reasoning. So the surviving subset is systematically the easy half, and the average over it is biased upward.

Worse, it breaks your sanity checks by inverting them:

• A truncated honeypot scores zero — which in a hallucination test looks like a pass. Your contamination detector now reports "clean" precisely because it failed.
• A truncated frontier model can score below your deliberately-weak control model — which fails your grading-sanity check, and reads as "the benchmark is broken" rather than "the harness truncated."

So above a threshold — I use ten percent — the run publishes nothing. It produces a bug report about the harness instead.

Two more ways to get an empty answer and a full invoice

Truncation was the first mechanism I found that produced a blank answer at full price. It was not the last, and both of the others are worth knowing before you go looking, because a reader who fixes only the token budget will still hit them.

Don't index the first content block. Responses from reasoning-capable models are a list of blocks, and the thinking block can come first. Reaching for element zero and calling it the answer yields an empty string on exactly the prompts that reasoned hardest — silently, with the full bill, and with a plausible-looking zero in your results. Filter the list by block type and concatenate what you actually asked for; never index by position.

Check that the field you're reporting exists. I built a scorecard column for reasoning-token share, and it read zero for every row. Not "sometimes zero" — structurally zero, because the API surface I was calling returns input, output and total tokens and does not break out a reasoning count at all. The column would have printed "0% of the budget went to reasoning" for a run that spent essentially all of it thinking. This is the entire thesis of this post in one field: it didn't error, it didn't warn, it produced a confident number that was wrong. Assert that a metric is non-zero somewhere in your test suite, or you will publish an artifact of your own plumbing.

Failure #2: I Couldn't Pin the Judge's Version

An LLM judge needs to be reproducible. If the judge silently changes between March and September, your longitudinal comparison is measuring judge drift and calling it model improvement.

So I went to pin the judge to a dated model snapshot, and found I couldn't — but not for the reason I expected. The barrier isn't the routing tier: globally-routed endpoints don't categorically reject dated identifiers. The problem is that the newest model releases simply have no dated identifier to use. A model like anthropic.claude-opus-5 has no date component in its ID at all — there is no opus-5-20260101 variant to pin to. The models that did expose a dated snapshot identifier were older or weaker. The one whose snapshot resolved was a small fast model, too weak to judge expert-domain answers. Test the exact identifier you intend to pin against the actual API rather than reasoning from the routing tier — the barrier to pinning is in the model catalogue, not the endpoint configuration.

That leaves three honest options: accept an unpinnable alias, accept a weaker but pinnable judge, or don't have a judge. I took the first, behind an explicit configuration flag so it's a visible decision rather than an accident, plus a nightly drift check as the compensating control.

With one caveat I should state now rather than let you discover in the next section: that drift check is not yet a control I've demonstrated works. Its golden baselines were authored before a single live judge call had succeeded and have never been calibrated against one, and the global tolerance it was supposed to enforce was dead code until quite recently. So the honest position is that I accepted an unpinnable judge on the strength of a compensating control that is itself unproven. Given the next thing I'm about to describe is a compensating control that had never once run, I'd rather flag the pattern than repeat it silently.

A related honesty problem: the API response echoes back the identifier you sent, so when you call an alias, nothing downstream can record which underlying version actually served the request. Any score built on an alias is attributable to a name, not a version. State that limitation in the report rather than implying a provenance the platform can't give you.

Failure #3: The Compensating Control Had Never Run

This is the one that should worry you most, because it's not about models at all.

Remember the nightly drift check — the control that made the unpinnable judge acceptable? Weeks later I discovered it had failed on every single run since the day it shipped.

The mechanics were mundane. The config flag was correct in the deployed configuration. The parser read it into the config object correctly. The judge constructor accepted the parameter correctly. And the one production call site — the nightly job — constructed the judge without passing it.

Several hundred tests passed the entire time. The overwhelming majority never touch the judge at all — and crucially, every test that did exercise it constructed it directly, so not one of them ever crossed the seam that was broken.

Concretely, what I changed:

• Replace per-call-site opt-ins with a single construction path. A keyword argument defaulting to the unsafe value is a defect generator — every new call site is a fresh chance to forget. A from_config(cfg) classmethod becomes the only production path, and a grep asserts the raw constructor has zero production callers.
• At least one test drives the entry point — the CLI, the job's main(), the request handler — with the deployed config values, not a fixture.
• Prove the regression test catches the bug. Revert the fix, watch the test fail with the production traceback, restore. A test that passes both ways tests nothing.
• Annotate the config file itself — "this only takes effect via X" — because the next person reads the ConfigMap, not the constructor.

A related smell surfaced in the same pass: a same-family cross-check compared the candidate's model family to the judge's, but the judge was constructed with an empty string family. The comparison was therefore always false and the check could never fire. When an unknown value would skip a safety check, raise instead of defaulting. Assuming "different" is the unsafe direction.

Failure #4: Grading Claude With Claude

A subtler validity problem. If your candidate model and your judge model come from the same family, you have a conflict of interest baked into the measurement — shared training lineage means shared blind spots and a plausible same-family preference.

My harness detects this and records judge_conflict = true, which means no rubric score is publishable for that pairing. Correct behaviour, and it also means I currently can't publish rubric scores for the models I most want to evaluate. The fix is a secondary judge from a different vendor family, and there's a useful practical note: several non-Anthropic model families were reachable through the same private network path with no firewall change, which made cross-vendor judging an afternoon's work rather than a security review.

One gotcha I'd flag before anyone builds this: my code sends the reasoning-effort parameter unconditionally, and the check for whether effort is supported introspects the client adapter, not the model. A non-Anthropic model that rejects that parameter would make every secondary judge call raise. Day-zero verification item, not a day-thirty one.

The People Problem Is the Real Blocker

Everything above is engineering. The actual critical path is human: who says the ground truth is correct?

Of my 33 challenges, 24 need sign-off from someone qualified in that domain. I can author a plausible answer key for domains I know well; for the rest, an unreviewed answer key means the benchmark measures my misconceptions and reports them as model errors.

Two things made this tractable.

Bind approval to a content hash

Record the approval against a hash of the exact text the reviewer read — prompt, ground truth, rubric and assertion specs together — not against the challenge's ID.

An ID-keyed approval survives later edits. Someone can get sign-off on benign wording and then change the substance underneath, and the gate still reports the item as reviewed. Hash-binding makes the approval lapse automatically when the text moves; nobody has to remember to revoke it. Because the hash covers the assertions too, changing a grader invalidates the approval as well — not just changing the prose.

Four details that make the difference between a real gate and a decorative one:

• Approval at a different hash is stale, and stale is distinct from never reviewed. Stale needs a quick re-read; telling a scarce expert they're starting from scratch wastes the resource you least want to waste.
• Append-only. A reviewer changing their mind adds a record. Editing in place erases the audit trail that is the entire point.
• A corrupt or missing store reads as empty — which approves nothing and keeps the gate closed. Failing open here silently publishes unreviewed material.
• changes_requested must not satisfy the gate. Counting a request for fixes as sign-off is the worst available misreading of a reviewer's intent.

Remove the friction, because friction is what actually stalls reviews

My first design had reviewers approve by editing a file in the repository and triggering a deploy. Zero reviews happened. Not because anyone disagreed — because a domain expert with a day job will not learn your git workflow to leave a comment.

Capture the decision as data through a simple web page and have the gate read that. Same for granting reviewer access: if adding a reviewer requires a config change and a rebuild, you become the bottleneck for your own review process. Make it self-service.

And scope reviewer rights per domain, separately from admin rights. Reviewing content and spending a budget are different authorities, and an admin is not automatically a domain expert.

Publishing without sign-off — the honest version

I was asked whether we could just run everything without waiting for reviewers. The answer is yes, and the distinction matters: there's a flag that bypasses the refusal, emitting watermarked artifacts with every blocking reason printed on them. A run that says "nobody has checked these answer keys" is honest and useful.

What must never happen is writing approval records on a reviewer's behalf. The record stores a reviewer's email and role, so a fabricated entry attributes a professional judgement to a named person who never made it. Bypassing a gate is a decision. Forging the gate's output is something else entirely.

What Thirty-Three Challenges Can and Cannot Tell You

Everything above is about not fooling yourself with a broken harness. This section is about not fooling yourself with a working one, and I put it late because it's the part I'd most want a sceptical reader to hold me to.

Run every challenge more than once. I use three repeats and report mean, standard deviation, and the share of challenges that passed on every attempt. Single-shot evaluation of a non-deterministic system is not measurement, and the failure is subtle: if you run once and average, per-challenge consistency computes to a flawless-looking 1.000 from the least possible evidence. My harness now refuses to report consistency at all when repeats equal one, because a fabricated perfect score is worse than a blank.

Then state the interval, not just the number. Thirty-three challenges, of which thirty are scored, at three repeats each. I compute a bootstrap ninety-five percent confidence interval per model and suppress any difficulty or capability flag whose interval is wider than 0.3 — because at that width the flag is describing noise. Which leads to the sentence that belongs in every internal benchmark write-up and is missing from most of them:

And the framing that keeps the whole thing honest in a governance conversation: thirty-three challenges cannot represent an organisation. This is a signal, not a certification. It is enormously more useful than a public leaderboard for our purposes, and it is still a narrow instrument.

What This Cost, and What It's Worth

The failures came in a chain, and this is the part I'd most want to convey: each one only became visible after the previous was fixed. A judge configuration bug hid a storage-permissions problem, which hid a filesystem-semantics problem — a storage layer that doesn't implement atomic rename, so the write-to-temp-then-rename idiom throws instead of working — which in turn hid the truncation problem. Every failure masked the next, and none of them announced itself.

I have to be straight about where that leaves the project, because it's the most useful number in this post. I do not yet have a saved score for any model. Two full sweeps ran every challenge to completion and then lost one hundred percent of their results at the final write, on a permissions fault that only existed on the real output path. Every figure I've produced so far is fixture replay: it proves the harness works and says nothing whatsoever about any model.

And "four failures" was me being kind to myself. Counting honestly across five versions of the harness it's closer to fourteen — four in the chain above, four more in the sweep that lost its results, and six found in review of the version after that. If you are budgeting for this work, budget for fourteen. A first evaluation run is not "does the model score well." It is "does my harness work at all," and you will find out considerably more than once.

The good news on the human side: 24 challenges need external sign-off, at roughly twenty to thirty minutes each. That's three challenges per reviewer across eight domains — about an hour to ninety minutes each, once. Framed that way, people say yes. Framed as "help me review the benchmark," they don't. (The total is eight to twelve person-hours; the thing that makes it tractable is that no individual is asked for more than an afternoon's fraction.)

One sequencing tip: start with the highest-risk domains, the ones where a wrong answer key does the most damage. If you made a systematic authoring mistake, it shows up in the first three challenges rather than after seven more people have spent their hour.

The purpose of an internal benchmark isn't to beat a leaderboard. It's to make a migration decision defensible — to be able to say "on the tasks our practitioners reviewed as representative, here's the difference, measured under these exact conditions, at this sample size, with these known limitations — and here is where the difference is too small to call." That's a sentence you can take to a governance review. "It felt sharper" is not.