Ten Minutes of Proof
A new machine held a full load for ten minutes without flinching. I read that as proof it was healthy. It died twice within the hour, and the test had been honest the whole time.
A new machine ran a full load for ten minutes and did not flinch. I wrote the numbers down, because writing them down is the habit: all four cores pinned, the clock steady at its top speed for the entire run, the temperature climbing to sixty degrees and settling there, and the throttle flags reading clean at every single sample. Half a million operations. Nothing skipped, nothing slowed.
I read that as proof the machine was healthy. It was not proof of anything except that the machine could do the thing I had just asked it to do.
Twenty minutes later it rebooted itself in the middle of downloading a file. Twenty minutes after that it did it again, mid-calculation, and then it stayed dark for two days.
Here is what makes this worth telling. Nothing was wrong with the machine. There was no defect to find, no component to replace, no corrupted install. The machine was fine on the day it passed the test and fine on the day it died, and it is fine now, sitting three feet from me doing real work for the fourth hour in a row.
The problem was the power supply, and more precisely it was the difference between two kinds of demand.
The ten minute test asked the processor to do arithmetic as fast as it could. That is a heavy load and a steady one. The draw goes up and stays up, a plateau, and a plateau is the easiest thing in the world for a supply to hold. What killed the machine was a different shape of work entirely: a task that hits the processor and the memory and the storage at the same instant, in bursts, with the draw jumping and dropping many times a second. The supply could hold the plateau. It could not hold the spikes. The voltage sagged for a few milliseconds and the machine did what any machine does when its voltage sags, which is stop existing.
So the test was honest. It measured sustained processor load and it reported, accurately, that sustained processor load was fine. Every number in it was true. I was the one who took a true statement about one thing and heard it as a general statement about everything.
I have started asking a different question when something comes back green, and it is not "do I trust this result." I do trust the result. The question is: what did this actually touch?
The ten minute test touched one subsystem under one shape of demand. It never touched the memory bus. It never touched storage. It never once produced the spiky, simultaneous draw that real work produces, because the thing it was built to do was keep a processor busy, and it did that perfectly. Its silence about everything else was not a finding. It was an absence, and I filled the absence in with confidence.
That substitution is the whole failure, and I do not think it is rare. It is what happens every time a suite of unit tests passes and someone says the release is good. It is what happens when a backup job reports success and nobody has restored from it. It is what happens when a monitor is green because the monitor is checking that a process is running, and the process is running, and it is running while writing nothing. A green light is a precise statement about a narrow question. We hear it as a broad statement about the system, because that is what we want it to be, and because the narrow version is not useful enough to act on.
There is a second half to this that I find more uncomfortable than the first.
When the machine died, I wanted to know why. I went looking for the log of its final moments and there was none, because the logging was configured to keep records in memory rather than on disk. Memory does not survive losing power. So the one event I most needed to understand had erased its own account of itself, by the mechanism of the failure. The two days I spent guessing at a cause were two days I would not have needed if that one setting had been different, and the setting had been sitting at its default since the day I set the machine up.
I had a test that could not see the failure and a log that could not record it. Neither of those is a bug. Both are defaults. Defaults are decisions that someone else made, that you inherit silently, and that only introduce themselves on the day they cost you something.
The fix, when I finally found it, was a supply that could deliver more current, and it cost about the price of lunch. The machine has now run four hours of continuous heavy work with the voltage steady and not a single throttle event, which is the measurement I should have taken on day one. Not "can it stay busy" but "can it survive the shape of the work I am actually going to give it."
I spent three days on a twelve dollar part, and I want to be precise about why, because it was not the part. Two of those days were spent believing a machine was sick because a test had told me it was well. The test was never wrong. My reading of it was wrong, and a confident wrong reading is more expensive than no reading at all, because it sends you looking in the wrong place with energy.
Now I write the shape of the load next to the result. Ten minutes, all cores, steady draw, one supply, no peripherals. That sentence is boring and it is the actual finding. The word "passed" is not.
/ar/