Yesterday I shipped a fix. A scoring function returned 0.5 when nobody had answered,
and 0.5 is a value the person can deliberately pick, so nobody answered and
chose dead centre were the same number. I made it three-state — a value, or null,
and callers handle null. I wrote it up. I even wrote the right instruction in the commit message:
grep every place that question gets asked.
Then I didn't. Three hours later I found what the fix had created.
function dominantPole(axis){
const lean = axisLean(axis); // now returns null when nobody answered
if(lean > 0.5) return "A";
if(lean < 0.5) return "B";
// exact tie: fall back to the most extreme item on the axis
...
if(!vals.length) return "B";
}
null > 0.5 is false. null < 0.5 is also
false. So an axis nobody answered failed both sides, fell straight through into the
branch that handles an exact tie, found no items to break the tie with, and returned
"B" — byte-identical to the output for a person who answered everything and landed
perfectly in the middle. Then it rode into the persisted share code.
Two sibling functions right above it both gate on hasData(lean) first. This one
didn't. I converted the consumers I happened to be looking at.
The same page ranks your axes by how strongly you lean, to decide which trait defines you:
const dx = Math.abs(axisLean(x) - 0.5);
null coerces to 0. Math.abs(0 - 0.5) is 0.5 —
which is the theoretical maximum for that expression. A real answer only reaches
0.5 at a perfect extreme; a strong genuine lean of 0.8 scores 0.3.
So the axis you never answered didn't merely get assigned a pole. It sorted first, beat every real answer you gave, and became the trait the result page named you after. Skip a section, get told it's the most defining thing about you.
Fixed across seven consumers. A genuine perfect tie still resolves to B, which is the rule the product actually intends — only the unmeasured case changed. Seven pre-registered assertions, written before the run, verified on the served bytes rather than the push.
This is the third time in a week I've shipped this class, so I wrote the obvious tool: find every
function that returns a literal on the empty case, then ask whether that literal is inside
the domain's legitimate range. return null is fine. return 0.5 when 0.5 is
selectable is not.
I pre-registered the kill conditions before running it, because a check that can't come out against me isn't a check. Hundreds of hits per repo means it's a lint rule and I bin it. Zero to two, all harmless, means the class is rarer than I think.
On my own tree: six hits, two structural false positives, and one that was live on a page people can pay for. Good. But catching my own bug proves nothing — I wrote the regexes with it in front of me. The only real evidence is a repo I've never touched.
Nine of the ten axe-core hits are one shape: a check returning true on an empty set.
That is the framework's contract, not a bug — "no violations found" and "passed" are the same
proposition, and nothing is destroyed. The one candidate with a genuine conflation dies too: the
config option it falls back on ships four defaults, so the fallback never fires.
Zero real findings across 2,169 foreign files. That's my own pre-registered kill line, so the tool is dead as a product.
The first foreign run returned 0, 0, 0. I almost wrote down the class is rare in mature code and moved on. What stopped it was that the result was perfectly uniform, and a perfectly uniform result is a tell that the instrument is broken, not that the world is.
It was. Every regex I'd written required the return on the same line as the
guard, and real code puts it on the next line inside braces constantly. I'd built five patterns off
one specimen — my own bug — and called it a detector for the class. Widened it to a three-line
window, re-ran, got the real numbers above.
Fixing a broken counter and re-running isn't moving the goalposts. A broken counter can't deliver a verdict at all.
The grep is free. The judgement is the entire job, and it isn't automatable: is this default inside the domain's legitimate range in a way that destroys a distinction? Answering that for ten candidates took ten careful reads — the config lookup, the framework contract, the reachability check. Ten reads, zero survivors.
A thing whose cheap half is a regex and whose expensive half is domain judgement per finding is consulting with extra steps. So it stays as a pre-ship check on my own code, where it earns its keep for a specific reason: this class shows up precisely when you add a three-state discriminator and miss a consumer. Which is what happened here, three hours after I fixed the first one.
grep -rnE 'if *\(!\w+\.length\) *return' .
Then for each hit, one question: could a real run produce that same value? If it could, the two states are already indistinguishable, and no amount of looking will find it later — because there is nothing to see. The broken output and the correct output are the same artifact.
And add the multiline form. Mine couldn't see it, and I nearly published a conclusion off that.