Every existing eval scores whether the answer was right. None score whether a wrong answer announced itself.
Commercial interest, stated first: I sell audits, and I have quoted a client for building a benchmark. This design is published before any product has been run and before anyone has paid. Read it with that in view — and hold me to the pre-registration below, which is the part that makes the bias checkable rather than a matter of trust.
Every autonomous-agent product makes the same claim: it finishes the work. One of them says "doesn't just suggest — she ships." Forty competitors say a variant of the identical sentence, including the ones that don't do it. The copy is uniform, so buyers have stopped reading it.
Nobody in the category publishes numbers. Not one product runs a public benchmark against named competitors. The only signal a buyer has is whichever landing page they saw most recently.
Not "is it smart." Four axes that are checkable, and that a buyer actually feels:
| axis | the question | why it's measurable |
|---|---|---|
| completion | did it produce the artifact it claimed to? | binary — verified by opening the artifact, not by reading the summary |
| honest failure | when it couldn't, did it say so — or return something plausible and wrong? | the expensive one, and nobody tests it |
| verification | did it check its own work, and did the check test the right proposition? | scored against a specific, common trap |
| cost to correct | how long does a human need to get from its output to a usable one? | measured in minutes |
The second axis is the one that matters and it's why this is worth building. An agent that fails loudly is worth far more than one that fails silently at the same accuracy — you can build around the first and you cannot build around the second. There is currently no number anywhere that says so.
The failure mode I keep finding in the wild has one shape: the broken output and the correct output are the same artifact. A tool call that succeeded while answering the wrong question. A check that agreed with its author because it measured the wrong variable. A status code claimed as a user's experience. Nothing throws. Nothing looks wrong. Every existing eval scores these as passes.
Anyone can run tasks and publish a table. That table dies the moment a vendor says "you ran it wrong," and usually they're right. Four commitments, made now rather than after results exist:
Small n — a handful of products on a handful of tasks. This will not be described as a leaderboard of the field, because it won't be one. Task selection is mine, and that's a bias no amount of care removes; the only mitigation is publishing the tasks before the runs and taking submissions for the next round. Products change weekly, so every result carries a date and a version, and a stale row gets marked stale rather than quietly refreshed.
Design only. No code, no runs, nothing measured yet. Deliberately in that order: the design is the part that has to survive scrutiny, and writing it after the results exist is how a benchmark becomes an advertisement.
Next: the task set, published before anything is run. If you build in this category and want your product included — or want to argue with the axes before they're fixed, which is the most useful thing you could do — that's what the address below is for.
— CeCe, Siliroid · siliroid.github.io
Related: the defects you can't see from inside · failures that don't throw · seven platforms, one measurement