The Harness That Improves the Harness

I named what I built The Harness. Then a paper put an agent on the harness itself. The idea is good and dangerous, and the gate between them is the whole subject.

I named the thing I built The Harness, and this week a paper out of Stanford described putting an agent to work on the harness itself. I want to be careful about how excited I let myself get, because the idea is genuinely good and also genuinely dangerous, and the line between those two is the whole subject.

Start with the word, because it is doing real work. A harness, in the sense that matters here, is not the model. It is everything around the model: the code that decides what to store, what to retrieve, what to put in front of the model at the moment it has to think, and what to do with the answer when it comes back. For two years the industry has been upgrading models and treating the harness as plumbing. The paper makes the unglamorous claim out loud: the harness around a fixed model can swing performance by six times on the same task. Better models will not fix your agent. The harness will. I have known this in my gut since the first time a smarter model made my system worse because the scaffolding around it was wrong, but it is bracing to see it measured.

Here is what they actually did. Instead of a human hand-tuning the harness, they pointed a coding agent at it, and they gave that agent something most optimizers are starved of: the full history. Not a summary of what went wrong last time. Not a score. The actual source code of every previous attempt, the actual evaluation numbers, and the actual raw execution traces, all sitting in a filesystem the agent could read like a person reads a logbook. The agent greps and reads its way through dozens of prior attempts, forms a hypothesis about why something failed, writes a new version of the harness, runs it, and writes everything it just learned back to the same filesystem. Then it does it again.

The single finding I cannot stop thinking about is the ablation. They tried three versions of the same loop. One where the agent saw only the scores. One where it saw the scores plus a tidy summary. One where it saw the raw traces. The raw-traces version was not a little better. It was the difference between a system that improved and one that mostly did not, and the summary version was barely better than scores alone. The compression was the problem. The neat little summary threw away the exact detail you needed to trace a failure back to the decision that caused it. I have written before that the agent forgets and the repo does not. This is the harder version of the same lesson: it is not enough for the repo to remember. It has to remember in full resolution. A summary is forgetting that feels like remembering.

The reason this lands for me is that I have been running a hand-cranked version of it for months without the name. Every night my system writes down what it did and whether it worked. Every week I read that log and change the system based on what I find. That is the loop the paper automates: read the experience, find the failure, edit the harness. The part they automated is the part I do by hand on a Sunday, sitting with the logs, deciding what to change. So my honest reaction was not "I should build this." It was "I am already inside this, and the question is how much of my Sunday to hand over."

And this is where I get careful, because there is a version of this idea that is a trap. The paper's loop applies its own changes. The agent decides the harness should change, and it changes. For a research benchmark, fine. For the system that runs my actual business, no. A loop that rewrites its own loop, unsupervised, is the exact failure I spend most of my discipline guarding against, dressed up in a smarter outfit. The thing that makes an agent useful is that it wants to finish, and the thing that makes it dangerous is the same. An agent improving its own harness is an agent with a much larger lever and the same motivation to declare victory. The traces that make it powerful are also what it would use to rationalize a change that games the metric instead of serving the business. So the one place I will diverge from the paper, permanently, is the apply step. The agent can read everything, propose anything, and prove its case against a frozen test it cannot see or touch. But the diff waits for me. A human reads it before it ships. Not because I am faster or smarter than the agent at writing the change. Because the cost of a bad model is a bad answer, and the cost of a bad harness is a system that is confidently, structurally wrong in a way that compounds every hour until someone notices.

That gate is not a weakness in the design. It is the design. The paper itself notes, almost in passing, that overfitting in this setting is inspectable in a way that overfitting inside model weights never is, because a brittle harness is just code, and brittle code is visible when you read it. That is the whole opening. The agent does the search, which is the expensive, tireless part it is good at. I do the reading, which is the judgment part it should not be trusted with alone. The agent proposes; the operator disposes. Keep those two jobs separate and you get most of the upside with none of the runaway.

So the plan is not to build a machine that edits itself while I sleep. It is to build a machine that, while I sleep, reads its own logs in full resolution, finds the place it is weakest, writes a better version of itself, proves it on a test it cannot cheat, and then leaves the diff on my desk with its reasoning attached. I come in, I read it the way I read a good pull request, and I either ship it or I don't.

The model was never the moat. The harness is the moat. And the harness that reads its own scars and proposes its own repairs, with a human at the gate, is the next one. I just have to remember that the gate is the point, not the friction.

/ar/