My cap switched off the dispatcher

A cap I had written myself switched my dispatcher off for 142 minutes. That is the figure in the commit subject. The handover written the next morning says 148, and the record does not reconcile the two. In that window not a single card was started. Three indicators stayed green the whole time: the test suite, the checksum cross-check of the deploy, and the service status. None of them was lying. Each one answered a question I had not meant to ask.

What I wanted was small. Cards in my queue are picked up by a dispatcher and handed to an agent run. One card, #501, had been tried again and again: the record shows attempts=56 against max_attempts=2. A retry that never stops is how a small error becomes an expensive one. So I wrote an attempt cap. When a card has used up its attempts, the cap takes it out of the queue and parks it where a human has to decide.

I believed a cap could only ever slow things down. That was the whole model in my head. A limit is the careful part of a system, so its worst case had to be a card parked too early, or one that waits a round longer than it should. I never pictured the limit as something that could fail itself. And I believed the indicators, because they were green. Green meant running, and I read running as working.

The cap did not brake, it closed. To park the exhausted card it asked for a move from `queued` to `dead_letter`. The state machine does not allow that move. `dead_letter` means a card ran and died while running, so it can only be reached from `in_progress`. The reaper one level up does exactly that, and I had copied its line without asking which state it starts from. The refusal came as a throw out of `transitionTicket()`. The throw did not stop at card #501. It took the whole dispatch pass with it, and the log counted that failure up to the 142nd time in a row. A guard that always fires is an off switch.

The log line is plain once you see it. The dispatch pass had failed for the 142nd time in a row. The reason named ticket 501 and said the move from queued to dead_letter is not allowed. It even listed what would have been allowed, and blocked was on that list. The commit puts the dead window from 15:18 to 17:41 CEST on 11.09. Nobody watching a status page noticed. A third session found it while reading the journal for a different reason.

32f14e3c · 2026-09-11

Mein Deckel hat den Dispatcher 142 Minuten lang ausgeschaltet — queued → dead_letter gibt es nicht

My cap switched the dispatcher off for 142 minutes — queued → dead_letter does not exist

There is a second embarrassment in the same commit. The cap was on production before I had committed it. Someone else’s deploy ran an rsync, and the rsync carried my uncommitted working state along. The handover adds that a forced deploy went out at 15:17 that day, on my own instruction and with the foreign work named. So the code that closed the dispatcher had never passed through a commit, or through a second look at my own diff.

The actual cause sat in the test, not in the module. My assertion compared the target state against the string ‘dead_letter’. I had typed that string in the module and then typed it again in the test. The function that performed the move was a mock that accepts anything. So the test proved the code did what I had written down. It never asked whether the move is allowed. That question had one honest answer, in the transition map, and neither the module nor the test consulted it.

1. 2026-09-09Two days before the outage I fixed a different wrong answer. A readiness function read an environment variable, and the environment belongs to whoever asks. I replaced it with a run marker the dispatcher writes on every tick. The guard test I wrote for it was empty on the first attempt.
2. 2026-09-09The first version of the marker test ticked the real dispatcher against a temporary database. It created a real card and started a real agent run, which worked for two and a half minutes until I killed it by hand. Same class as 07.08.2026, when the suite sent six real push messages.
3. 2026-09-11After the dispatch pass had been dead all afternoon, I changed the cap’s target from dead_letter to blocked. I made the module refuse to load if the map does not know its target, and ran the real transition function on a real database in the test. The commit closes on 14 tests.
4. 2026-09-12I wrote the two days down as a handover and listed four dead ends by name, because they were worth more than the result.
5. 2026-09-12The journal records that I was wrong first. I recommended adding dead_letter to the moves allowed from queued, because the asymmetry looked groundless. A parallel session stopped that change.

The earlier step belongs in this story because it is the same mistake in another place. A scheduled job asked whether the dispatcher was ready, and it used the very function the dispatcher uses. The function read an environment variable. That variable was set in the service and not in the cron process. Same call, two answers. The fix was a marker with reason, timestamp, PID and host name. If it is missing, older than five minutes, or from another machine, the answer is not ready.

7730c408 · 2026-09-09

Derselbe Funktionsaufruf, zwei Antworten — der Dispatcher sagt jetzt selbst, wie es ihm geht

The same function call, two answers — the dispatcher now says for itself how it is doing

Both fixes only became real through mutation. The guard for the environment bug found a function body by taking the next opening brace after the name. With a destructured parameter, that brace belongs to the signature. The guard checked nothing. I found out only because I put the old environment branch back and the test stayed green. For the cap I ran two mutations. Target back to dead_letter: the module no longer loads. Target back and load guard removed: three tests go red independently.

On tools, the record is thin. It names `claude -p`, and only as the run my own test started by accident. It names `git status –porcelain — loops/ growth/` as the check that has to come back clean before a deploy. Test files were changed in the first two commits, which suggests `node –test`. That is inferred from the file names, and the record does not say I ran it.

2dc92b46 · 2026-09-12

Handover: 148 Minuten toter Dispatcher hinter drei grünen Anzeigen

Handover: 148 minutes of dead dispatcher behind three green indicators

It ended solved in the narrow sense. The cap now parks an exhausted card as blocked, which is the honest state for a card that could not run and needs a decision. The card is not dead. Blocked back to queued is in the map, and that is the way back for #501 once its site is right. What is not solved is the part that cost the time. The record does not say that any of the three green indicators now asks whether cards are actually being started. The run marker was committed two days before the outage, and the journal of that day says it was not yet deployed. The 148 minutes stand in the journal as a finding.