The Zombie Signal

Two alarms that kept screaming after the fire was out, and why an alert that cannot turn itself off is the dangerous kind.

Every morning a short report lands in my inbox before I am awake. It reads the night's data across the restaurant and the systems behind it, and it tells me what is worth my attention. Most mornings it is quiet, which is the entire point. The mornings it is not quiet are the ones I built it for.

For about a week it told me the same two things were broken. One: that I had not received a single vendor invoice in eleven days, which for a place that orders food every few days is somewhere between implausible and alarming. Two: that the system which links each customer to what they order had gone dark and stopped doing its job. Two red lines, at the top of the page, every single morning.

Both were false. Not slightly off. Completely, already-fixed false.

The invoice line was screaming about a gap that had closed days earlier, the moment a new piece of automation quietly started pulling those receipts in. The data was flowing fine. But the alarm had been written down once, as a fact, on the day the gap was real, and nothing ever went back to ask whether it was still true. It just kept getting repeated, the way a rumor outlives the thing it was about.

The second was subtler, and honestly worse. That linker only does its work once a month, when a particular export arrives. The alarm was set to panic if it saw no activity for ten days. So every month, in the three-week stretch between exports, it would wake up, see nothing happening, and declare the thing broken, even though nothing was wrong and never had been. I had built a smoke detector that goes off every time the oven is simply not on.

An alarm that only knows how to turn on

Here is why I care about this more than it seems to deserve. An alarm that can turn on but never turn itself off does not just annoy you. It quietly trains you to stop reading alarms. Each morning I learned, a little more, to skim past the red. And the whole reason that report exists is the one morning it is right, the morning something genuinely is on fire and I need to see it through all the noise I have taught myself to ignore.

The fix was not a cleverer alarm. It was making every alarm check itself, each morning, against what is actually true right now. If the condition has cleared, the warning resolves itself, silently. If it comes back, it re-opens itself, loudly. The state of the light is wired to the state of the world, not to something I typed once and forgot.

I carry a scar from the other side of this. I once had a real failure run silently for seventeen days because an alarm that should have fired never did. That felt like the opposite problem. It was not. It was the same disease wearing the other mask. In both cases the warning light had simply come unhooked from reality. Stuck on, stuck off, it does not matter which. A light that is not connected to the thing it watches is not a safety system. It is decoration that happens to tell the truth by accident.

So now I trust an alarm in proportion to one thing: whether it can clear itself. A warning I have to silence by hand is a warning I will eventually silence in my head. The ones worth keeping are the ones humble enough to admit, on their own, when they are wrong.

/ar/