The messenger died with the loop

I run agents at night so I can sleep. The deal is simple: the loop either does its work or it tells me it couldn’t. Silence is supposed to mean everything is fine. For nine nights, silence meant the opposite, and I had no way to tell the difference.

Eight of eight runs since August 24th had died, every one of them identically. Across 269 artifacts there were 0 grades. The loop had been dead for nine nights and the record was clean, because the run that should have shouted about it was dying on the same fault it was trying to report.

I believed a service that is running is a service that is working. I believed that if something had gone wrong, I would hear about it — that the absence of an alert was itself evidence. The messenger and the message were, in my head, two separate things. One could fail and the other would still reach me. That was the whole mistake.

The messenger was part of the same run. The run built a payload, an identity guard inspected it, found forbidden phrases in the report — the very report explaining why it could not post — and killed the run there. With the run died the notification that would have named the real cause: a config file the site pointed to was missing. So the alert that would have said “I am dead” was carried by the process that was dead. A messenger inside the same process as the message shoots itself. And the switch that turned the loop back on sat in three places, two of which knew nothing about each other.

The loop runs again. But the lesson is the shape, not the fix. A watchdog that only speaks on a state change can die quietly — because dying is not a state change it can report. What I ruled out along the way mattered as much as what I found: the working directory, the tool permissions, and a placeholder were each suspected and each cleared, so nobody has to chase them again.

The switch that sat in three places

That opening commit added 9 lines and named the 46 foreign files the one deploy command would carry. I was cutting the work down to exactly one thing — one command, one blast radius — because the cause lived two repos away from the symptom, and I did not want to move anything I had not accounted for.

Turning the loop back on was not one switch. It was three. The config in loops.json had to say enabled. That is necessary and not sufficient: the same shutdown had also removed the cron line. I counted nine run-loop lines left, and exactly the five paused ones were missing. An enabled flag with no cron line is a dead switch — set correctly, doing nothing, and in the cockpit the loop would look alive. What put me on the trail was the crontab; what briefly misled me was its own logfile, dated 01:20 that day, left over from the last run before the line vanished.

1. 2026-09-02Cut the recovery down to one deploy command and wrote down the 46 files it would drag along, so the blast radius was known before anything moved.
2. 2026-09-04Wrote up the trail from the conference-talk question to the cause two repos away, including my own botched commit of the opening.
3. 2026-09-04Recorded that the identity guard was destroying the very reports that named the cause, and that two frozen cards were not defects to be ‘repaired’.
4. 2026-09-04Pinned down the three suspicions I had ruled out — working directory, tool permissions, placeholder — so nobody would chase them a second time.
5. 2026-09-06Found the third place the switch lived, set the cron line back, and confirmed the loop actually runs.

The second missing place I did not find at all. A test did. It said 16 active loops need 29 runs and the cap stood at 27. With the cap too low, one scheduled run drops silently — and it would drop on the freshly reactivated loop, exactly where nobody would notice. A human looking at a green dashboard would have missed it; the arithmetic did not.

There were dead ends before any of this worked. The first was method: I measured against the code instead of against the real production database. Only when I read the live copy did the pattern show — 8 of 8 runs since August 24th failing with the same identity violation. And the switch itself was searched twice before it was clear which place actually governed it. My first grep, piped through head -20, cut the answer off before I could read it — a short list is not an identification.

$