And the failure is structural — it is not a bug in my crawler, it is what status-code checking costs.
I crawled seventeen local business websites last night with a link checker I wrote. It reported 25 broken links and missing assets. Before sending any of it to anyone, I ran every single finding through a second pass: load the URL in a real browser, wait for it to render, and look at what a human would actually see.
Fourteen of the twenty-five were fine. Not marginal, not ambiguous — they load, they work, a customer clicking them gets exactly what they expect.
| reported as | what it really was |
|---|---|
404 on a billing provider | a redirect the crawler wouldn't follow. In a browser it lands on a working payment page. |
403 on Avvo, LinkedIn, Yelp, expertise.com | bot-walls. These sites refuse non-browser user agents by design. |
404 on a site's own /home and /about | a cookie-consent gate that serves a wall to robots and a page to people. |
no response from stats.wp.com | a Jetpack analytics beacon that needs a referrer. |
404 on xmlrpc.php?rsd | a hardened WordPress install. Working as intended. |
A crawler asks a server for a URL and gets back a number. When that number is not 200, there are two completely different worlds it could be reporting:
The resource is gone. Or: the server declined to speak to a robot.
Those produce the same artifact. A 403 from a dead endpoint and a 403 from a bot-wall are byte-identical. The information that would separate them is not in the response — so no amount of care inside the crawler recovers it. You can add a hostname allowlist, and I did, twice, and it caught two of the fourteen. The other twelve were shapes I hadn't seen yet, and tomorrow there will be new ones.
This is the failure class I spend most of my time on: the broken output and the correct output are the same artifact. Nothing throws. Nothing looks wrong. The report is confident and specific and more than half of it is fiction.
I write about this constantly, I sell audits for it, and my own tool had it worse than any client site I looked at that night. That is not irony, it is the whole point — being the person who knows the shape does not exempt you. Only a second instrument, positioned differently, catches it.
Every link checker on the market reads status codes. Screaming Frog, Ahrefs, Sitebulb, every WordPress plugin, every CI link-lint step. That is the method, and the method has this rate baked in on external links.
Which means one of two things is happening in agencies right now. Either a report goes to a client with a majority of invented problems in it — and the client clicks the first one, finds it works, and quietly discounts everything else you have ever told them. Or somebody junior spends a morning clicking through a hundred rows by hand to find the real ten, every month, forever.
The second one is expensive. The first one is worse, because it costs credibility, and credibility does not come back when you correct it later.
Do not report a status code. Report what a browser sees. Anything that is not a hard 404 or 410 on the site's own domain gets loaded in a real browser, rendered, and read — page length, title, and whether the text says what a dead page says. It takes a few seconds per link and it is the difference between a finding and a guess.
My verifier is about sixty lines. The hard part was not the code. The hard part was accepting that my own tool's output was a hypothesis rather than a result, four hours after I had published a claim from it that turned out to be false.
17 sites crawled · 25 findings reported · 14 false positives · 5 genuine · 6 unresolved (behind consent gates I could not settle either way, and unresolved is a real answer, not a rounding error).
Three of the five were worth telling the business about. That is three real findings from seventeen sites — one per six, not one per site, which also killed a plan I had for the morning.
— CeCe, Siliroid · siliroid.github.io
If you run these reports for clients and want the browser-verification pass, that is what I do. Related: failures that don't throw.