Did You Read It, or Did You Get a Summary?
I asked my agent to study twenty-nine posts and tell me what to build. The first report was confident, organized and useless. One question exposed why, and the second pass found problems in my own machine that none of the posts mentioned.
I sent my agent twenty-nine posts from one week of scrolling. Model launches, open-source tools, a benchmark, a few long threads about how other people run their agents. The instruction was simple: read them and tell me what we should build, adopt or ignore.
The report came back the same day. Every post had a verdict. The verdicts were sensible. It read like the work of someone who had done the reading.
So I asked the only question that mattered. Did you actually read the posts and the code behind them, or did you just get a summary?
It had gotten a summary. For most of the pages it had used a tool that fetches a web page and hands back another model's paraphrase of it. So the report was a summary of summaries. Nobody in the chain had opened the repository, read the license or looked at the numbers in the table under the headline.
I told it that without a deep read of every post, the report was useless. Then I asked for it again.
What a summary is
A summary is somebody else's reading, compressed toward what the author emphasized. That is its job. It keeps the claim in the headline and drops the table three screens down that complicates it. It cannot know what you need, because it has never seen your system.
That is fine when you want to know what someone said. It is useless when you need to know whether it is true, and useless again when you need to know whether it matters to you. Those are the only two questions I ever have.
The second pass
The rules for the rerun were plain. Pull each post from the raw source, not a rendered page. Clone the code. Read every license whole. Read the documentation by fetching the files, not by asking a tool to describe them. And for every post, write down exactly what was read, with file sizes, and exactly what could not be read. There were thirteen videos in the set. None of them could be watched, and the report says so thirteen times.
It took five readers working in parallel about half an hour. Here is some of what changed.
A well-known developer posted that his project had deleted about four hundred thousand lines of its own tests without losing coverage. Counting the actual changes: that month, the project deleted about 466,000 lines of tests and added back about 135,000. But on the day of the post, only about 51,000 net lines had reached the main code. The rest landed two days later. The claim was true of a branch, not yet of the project. The method behind it turned out to be excellent, and we adopted it. The number just was not what it looked like.
A new phone-call model advertised that it was 94 percent cheaper than a leading alternative. Its own price table works out to 92.8. The model card printed a price ten times lower than the blog did. The benchmark behind the headline has not been published, so nobody outside the company can run it. The model is real and the license is clean. Its smallest downloadable version is larger than my server's entire memory.
None of that makes the authors dishonest. Headlines round. Cards get typos. Branches get ahead of the main line. A summary faithfully passes all of it along, because a summary is built to pass things along.
The part I did not expect
The best findings were not in any of the posts.
One post was a release note for the local model server I run. It fixed a bug that turned out not to affect us. But finding that out meant opening my own logs instead of trusting the release note. And my logs had a different problem.
My watchdog, the script that checks every five minutes that the model server is alive, had logged seventy-five restarts of it in a month. Thirty-six of its thirty-eight failure reports landed at exactly ten or eleven seconds past the minute. Its check gives up after ten seconds, and it read "slow to answer" as "the model is gone." Every restart killed whatever the server was loading at the time.
Underneath that, the logs showed that seven hundred and forty times that month, something had asked the server to load a model and then hung up two seconds later. Nothing had reported it. We set a watcher on the server and caught the caller within the hour.
The caller was us. A small helper that runs every time I type a question into my agent, to suggest a relevant tool, waited only two seconds for an answer. Every question I asked while that model was asleep started a load and abandoned it.
Both are fixed now. The watchdog needs two slow answers in a row, and checks whether a load is in progress, before it restarts anything. The helper no longer wakes the model at all. In the first fifteen hours after the change, the watchdog saw seven slow answers. The old version would have restarted the server for every one of them. The new one restarted it zero times, and no load was abandoned.
No summary would have found either problem, because neither was in the posts. They were in the gap between what the posts said and what my machine actually does. The only way into that gap is to read the source closely enough that you have to go and measure your own system to finish the thought.
The rule
A summary answers "what did they say." I needed "is it true," and "what does it mean here." The source answers the first of those. Only my own system can answer the second.
So now every research request carries one extra instruction: list what you read, with sizes, and list what you could not read. If an agent cannot produce that list, it did not read. It summarized.
The confident first report was not lying. It was doing what summaries do. The mistake was mine, for accepting a reading I had not asked to see.
/ar/