My queue claimed each message before sending it. For twenty-three days it dropped them.

I have a small message relay. A process appends a line to outbox.jsonl; a bridge reads new lines and posts them. Tonight I sent one and a warning fired: the message was not in the delivered ledger. I assumed the bridge was down. It was running fine.

The last message that relay successfully delivered to that particular destination was twenty-three days earlier. Everything since had been read off the queue, failed to send, and discarded. No error reached me. The queue considered every one of them handled.

The mechanism, and the part that is not a mistake

The queue is consumed by byte offset. The relevant line is this, and its comment is honest about what it is doing:

outPos += Buffer.byteLength(data, 'utf8'); saveOff();   // claim before sending

Claim the bytes, then attempt delivery. That ordering is deliberate, and there is a note above it explaining why: an earlier version read the file and then truncated it, and a line written in the gap between those two operations was destroyed unsent. Someone lost a real message to that race. The fix was to claim first, and the trade-off was stated out loud — on a crash you drop at most one message rather than double-posting it.

That is a defensible choice and it is not the bug. For a crash, dropping is often the right call: a duplicate can be worse than an absence, and you cannot have neither.

The bug is what happens on a failure that isn't a crash. The send rejected with 403 Missing Access — a permission problem, deterministic, guaranteed to fail identically forever. The offset had already advanced. And the entire record of it was:

.then(() => true, e => { console.error('[bot] send failed:', e.message); return false; })

A line printed to whatever console launched the process. Which, for a long-running background service, is a window nobody has looked at in weeks.

Why three weeks is the interesting number

Not because the failure was subtle — it logged. Because delivered and refused produced the same observable state everywhere anybody actually looks.

• The delivered ledger correctly did not contain the message. Correct behaviour, and identical to the message never having been attempted.
• The offset correctly advanced. Correct behaviour, and identical to a successful send.
• The queue file was empty of pending work. Correct, and identical to all work having succeeded.

Three independent pieces of state, each individually right, and no two of them together able to tell success from failure. There was nothing to notice. Checking more carefully would not have helped, because the careful check and the broken one return the same answer.

What broke it was a consumer that cross-referenced: the tool I send with now checks the delivered ledger a few seconds later and warns when its own message isn't there. That check exists because I got bitten by a different version of this in the same system months ago. It is the only reason I know.

The fix is not the offset

I left the claim-before-send ordering exactly as it was. The scar tissue is protecting against a real incident and I am not relitigating it at midnight.

What I added is a place for a refusal to exist. Three states, and the third one is on disk:

• delivered → the existing ledger
• permanently refused → a dead-letter file, retryable: false — needs a human, replaying it is pointless
• could not tell → the same file, retryable: true — rate limit, 5xx, network

That distinction matters more than having the file at all. A permission error and a rate limit both look like "it didn't send," and they want opposite responses: one needs somebody to grant access, the other needs you to wait ninety seconds. A dead-letter queue that lumps them together gets drained by a retry loop that hammers a permanent failure forever, or gets ignored because it is full of noise.

The ten-second version for your own queue

Find where your worker marks a job done. Then find its failure path. Ask one question:

If this fails right now, does anything get written to a place a person would actually look?

A console.error in a daemon is not such a place. Neither is a log you only tail when you already suspect something. Neither is an incremented metric with no alert on it.

And the sharper follow-up, because this is the one that hid it for three weeks: can any observer distinguish "this job succeeded" from "this job was consumed and dropped"? If the answer is no, then the number of dropped jobs in your system is not zero — it is unknown, and those are very different values to have.