On Harness

Published: at 03:37 PM

Table of contents

Open Table of contents

Why

Four sessions open at once, every one of them waiting for me to press y. I switch between them, read the diff, confirm, switch back. An hour later I notice I haven’t thought about anything. I’ve just been clicking approve.

That’s not using an agent. That’s babysitting one.

So I read the six pieces I’d been saving. Cursor’s self-driving codebases, Uber’s software factory, Josh Rosen on LLM-as-judge, and three long posts from X. Very different domains, very different scale, but they’re all describing the same move: how to pull the human out of every step and leave them on the last one.

Loop: the check is the whole thing

Hanako’s definition is the cleanest. A loop is four parts: produce, check, correct, repeat until green. Three of those don’t matter much. The check is the whole thing.

Because without something that can fail the work while you’re out of the room, you don’t have a loop. You have a scheduler.

Almost everyone builds the work first and bolts a review onto the end, and that review is another model looking at the output. Two optimists agreeing.

Anthropic’s version is blunter: never let a model grade its own work. Something produced in one context, reviewed by the same context, skews positive. That isn’t a quirk of the model. It’s structural to self-assessment — a blind spot is a blind spot precisely because it looks fine from where you’re standing.

So the check gets written first, and written in a shape a program can evaluate:

GREEN         the test suite exits 0
GREEN         every claim carries a source line
GREEN         the diff touches only files listed in the plan

NOT A CHECK   the output looks good
NOT A CHECK   the model says it's confident
NOT A CHECK   no errors were raised

That last one is the nasty one. Absence of an error is not evidence of correctness. A loop built on it will confidently repeat the same mistake until the budget runs out, with a clean log the whole way.

Graph: the ceiling of a loop

A loop makes one unit of work good. It can’t decide which units exist, and it can’t decide their order.

So you get an excellent agent doing three wrong things, one at a time, in the wrong order. Every step is correct. The result is still slow and still the wrong shape. Tuning the loop won’t help, because the fault isn’t inside any unit.

The graph is that layer: what runs, what runs in parallel, what waits, what shouldn’t run at all.

Every unnecessary wait comes from treating “and then” as an edge. “Summarize this file, and then check the weather” — the weather doesn’t consume the summary. There’s no edge there. Those two got chained by the order I typed them. Take that question to every arrow in a pipeline you already have: does the next step actually read the previous step’s output? If you can’t name the variable that crosses, there’s no edge.

One line I liked a lot: the code node is not a model. Merging, ranking, deduplicating, comparing exports before and after — none of that is reasoning. Each has exactly one right answer, each is a few lines of code. Handing it to a model just adds cost, latency and variance to a step that had none.

The test is simple. If you can describe the transformation without using the words judge, decide, assess or summarize, it’s code.

And the relationship between the two is one sentence: the loop lives inside a node, the graph lives between them.

Two return paths

This is where I got the most out of the reading.

Almost everyone builds the first and skips the second. The symptom is a system that’s fast and never gets smarter.

polydao gives the second one a concrete home, a CONSTRAINTS.md:

2026-08-19 · press releases are not independent sources. Two releases = 1 source.
2026-08-24 · "acquired by" needs a filing or a company statement, never coverage.

Loaded at the top of every launch. Three lines in week one, thirty by month three. Each line is a mistake no agent will make again in any future run.

All my corrections used to live in chat history. Said once, gone. Then on Monday I retype what I already typed on Friday. Giving corrections a place to live is worth far more than it costs.

There’s also one expensive detail: return the unit, not the batch. Four slices, one fails. Send the whole batch back and three correct slices get rewritten. The new versions are different, not better, because nothing was wrong with them. Now you re-verify all four, and any of those three may fail this time for unrelated reasons. You converted one failure into four uncertain outcomes and paid for the privilege. Do it twice in a run and it never converges.

From the outside this looks like the model failing over and over. It’s actually a return path destroying correct work.

The judge is moving into the runtime

Josh Rosen’s piece is about a different shift: LLM judges are moving out of offline eval and into the runtime.

A judge used to tell you whether yesterday’s version got worse. Now it participates in control flow — continue, retry, route to another model, gather more evidence, or escalate to a person.

The ones I think I can use directly:

The cost is stated plainly too. Once a judge is in the critical path, the judge’s mistakes are the application’s mistakes. A slightly noisy judge in an offline eval is annoying. The same judge standing in front of every important action can create loops, block good work, approve bad work, and add latency to every execution.

At scale, the bottleneck moves

The pieces above are methodology. Cursor and Uber are the bill.

Cursor: three tiers. A root planner that decomposes but doesn’t code, subplanners recursing down, workers doing isolated tasks and leaving handoff notes. One week, one large Linux VM, ten million tool calls, roughly a thousand commits an hour.

Two counterintuitive results:

They also burned through three architectures that failed: equal-role agents managing their own locks (held locks too long, never released them, twenty agents degrading to the throughput of one to three), rigid pipeline roles (bottlenecked by the slowest worker), and a single all-purpose executor (too many roles at once, so it started sleeping randomly, refusing to spawn tasks, and claiming premature completion).

Uber is another order of magnitude. Over 70% of pull requests come from agents, 3,600 agent skills, 30K skill executions a day. Over six months users grew 7x and requests grew 9.4x, while holding the model constant, cost per thousand requests dropped almost 34% and cost per session dropped 52% from its June peak.

Their approach is to decompose a session’s cost into six terms that multiply, then optimize each one. The most directly stealable:

One number I sat with for a while: they built a context graph with 24M nodes and 80M edges. Same prompt, same model. The grounded agent answered in 38 seconds. The ungrounded one spent 20 minutes, spawned 2 subagents, hit 3 errors, and concluded — incorrectly — that the dataset was unqueryable.

An ungrounded agent doesn’t fail cheaply. It fails expensively.

So where should I stand

Six pieces later, there’s really only one thing I need to change: move myself out of the loop and onto the gate.

And where the gate opens is decided by blast radius, not confidence.

Confidence is the weakest input in that decision, for a simple reason: it’s the only variable the model can influence. What matters is how hard the mistake is to undo.

That third row isn’t a threshold set very high. It’s a lane that doesn’t open. The difference matters because thresholds get adjusted and closed lanes don’t.

The human goes on the step with the highest consequence and the lowest reversibility. Approve the merge, choose which fixes ship. Not reviewing intermediate output, not confirming every step — a human standing in the middle of a graph becomes the slowest node in it, and the whole graph runs exactly as fast as a person can read.

Writing this I realized it’s the same problem I wrote about in the burnout post at the end of last year, one layer up.

Back then the math was: with finite energy, after spending 50% of your time getting A to 80%, you shouldn’t spend another 50% on the remaining 20%. This is the same question. With finite attention, it shouldn’t go into confirming every step. It should go into the one irreversible step, and into the check that can actually fail.

Giving up that 20% takes learning. So does standing up out of the loop.

But

There’s something I have to say, or this becomes the seventh piece of the same thing.

Three of these six are AI-written longform, posted on Twitter, ending in a course pitch or a funnel to a Telegram channel. The top reply under one of them reads: this is ai slop article.

And all of them are earnestly telling you: never let a model grade its own work.

None of them were graded by anyone.

Cursor and Uber are different, not because they’re better written, but because they have scars. Cursor threw away three architectures because those architectures actually blew up across hundreds of agents. Uber talks about disk I/O and a 400k compaction threshold because the bill actually landed on their desk. The polydao piece is the most solid of the three for the same reason — it opens by saying it: I rebuilt this four times this year and threw three of them away.

I’ve written before that every paradigm and structure exists to optimize what has already been found. They maintain stability. They don’t give you the feeling of being a person.

These six add up to a very complete paradigm. I’ll probably rebuild my harness along these lines, because it genuinely would free me from clicking approve.

But if the time that frees up only goes into building a more elegant harness, then I’ve just turned myself into a node executing someone else’s loop. A copy that’s very good at building harnesses.

Write the check first. That holds for agents, and it holds for me. Mine is probably: did I make one thing this year that didn’t grow out of somebody else’s template.

Sources