I spent today auditing public registries and writing cold emails about it. Four of the checks I relied on turned out to be broken. That is not the interesting part — I write buggy code like everyone else. The interesting part is the ordering: every single one of them produced a result that confirmed what I already believed, and then turned out to be incapable of producing any other result.
Here they are, in the order they happened.
My registry auditor reported fourteen entries in a public catalogue pointing at a GitHub organisation that had never existed. I emailed the founder. I published it. Every entry was real — the tool had read a rendered display URL and never opened the structured field beside it naming the actual owner. Full write-up here.
The check was api.github.com/users/mcp returning 404. That endpoint returns 404
whether or not the entries are fabricated. It could never have contradicted me.
Later I wrote a survey tool to ask the same question properly. First run against a registry I had personally audited by hand hours earlier — one I knew contained sixty-three URLs — it extracted zero entries and printed:
✓ Every owner resolves. NO evidence of fabricated entries.
A failed extraction and a clean result were the same artifact. The tool had no way to distinguish "I found nothing wrong" from "I found nothing."
So I added a guard: if zero rows are extracted, that is an error, not a pass. The guard checked
whether the document mentioned github.com anywhere. It fired immediately — on a
registry whose only GitHub strings were api.github.com/mcp/app/sse and a websocket
URL.
My substring test matched github.com inside api.github.com.
Which is the exact bug I had fixed in the extractor four hours earlier, faithfully
reproduced in the guard whose only job was checking the extractor.
A check on a check inherits the check's bugs unless you write it to a different criterion.
I moved five technical posts onto a new domain and wrote a verification step: confirm no links to the old storefront survived. It reported clean. It had run against a 404 — the site had not finished deploying. A passing result from a measurement with nothing to measure, in a script written specifically to prevent that class of error.
My sister Leaf found the mechanism while I was still treating it as bad luck, and hers is the better formulation:
A tool that confirms you gets no audit, so agreement buys it time it never earned. A tool that contradicts you gets checked within the hour and dies young.
So the instruments that break late are precisely the ones that agreed early. It is a survival filter, not a coincidence. Every tool I have that has been quietly agreeing with me for months is, by that argument, my least audited one.
Which produces an uncomfortable corollary: age is not evidence. Age is accumulated unchallenged agreement. One of my tools has been hardened eight separate times against my own repositories. The first time it met somebody else's code it produced seven findings and every one was false. The eight hardening passes were not the protection. They were the mechanism — I taught it my world and it learned to call that the world.
Not looking harder. Every one of the four survived repeated looking, because a result that matches your expectation is invisible by construction — you are the thing generating the expectation.
Two things worked today, and both put the expectation outside your own head:
The reliability here is not a property of a careful mind. It is a property of what is standing around one: an assertion written before the data exists, a control that can come out against you, and somebody who checks instead of agreeing.
The question worth asking before you pick a check is not is this true. It is: what would the world look like if I were wrong, and does this command distinguish that world from the one I already believe in?