On Units: The PR and Its Author

Published: at 02:33 AM

Table of contents

Open Table of contents

Seventy PRs

A Factorio clone went from an empty repo to 70 pull requests in five days. 67 merged. I wrote none of them.

In the same week a pixel-art portfolio got 59, a text-first one got 13 in an afternoon, and a desktop app that launches coding agents got about forty over two days. Same loop every time: an agent picks an issue and opens a PR, a second agent verifies it on its own, a third reworks what failed, I merge.

The first night it felt like cheating. By the second week it felt like running a small open-source project with a team I’d never met, all of whom type very fast.

That’s the post. Not that the loop works. It does. What happens to it once there’s a lot of it.

Day zero is easy

The loop I wrote about in On Harness holds up at the start. Write the check first. Never let a model grade its own work. Put the human on the gate, not in the middle.

On a new repo that’s almost free. There’s no history to break, no shared files to fight over, no one else’s tests to keep green. An issue goes in, a PR comes out, the verifier runs the suite and takes a screenshot. Most PRs pass on the first try because most of the code doesn’t exist yet.

issue → implementer → PR → independent verifier → rework (≤1) → merge

So the first week looks like a solved problem. It’s the easy part.

Time is the wrong axis

My first draft of this post was a timeline. Week one, week two, what broke when.

It read like a diary. “Things got harder” is true and tells you nothing. The project didn’t get harder because it got older. It got harder because it got bigger, and bigger has a shape.

Here’s what actually showed up, roughly in the order it bit:

conflicts     one scenario list touched by every PR — a conflict on almost every merge
capacity      ten agents at once, machine load 35, tests timing out as "flaky"
budget        a weekly usage limit hit mid-task; every agent stopped in the same minute
state         two container restarts; every agent gone, the shared rules wiped from scratch
environment   a dotfile in my home directory leaked into the test runs and broke every end-to-end run
requests      mid-flight changes: "one PR per issue", "stop that feature, research only"

None of these is about intelligence. The agents wrote good code the whole time. The failures were all coordination: who touches what, who checks what, how much runs at once, and what survives a restart.

Which made me stop asking “when” and start asking what the unit is.

The PR is the unit

Not the commit, too small. Not the week, too vague. Not the agent, which is just whoever happened to open it. The PR is the thing that gets checked, merged, reverted, and blamed.

So the PR needs a contract. Here’s the one I ended up enforcing, after the verifier kept finding ways around the softer version:

REQUIRED   one task, one issue, one PR — bundles got split mid-flight, twice
REQUIRED   a test that fails when the fix is reverted, shown, not claimed
REQUIRED   the evidence names the commit it ran on
REQUIRED   the real binaries ran against a stub of the remote service
REQUIRED   screenshots for layout, interaction tests for behaviour

NOT EVIDENCE   "tests pass" on a branch that predates the merge it depends on
NOT EVIDENCE   a suite that silently skipped its browser tests
NOT EVIDENCE   a test that rebuilds the fix inside itself

That last line cost the most. On one reader module the verifier broke the real code 21 different ways, and 8 of those still passed the PR’s own tests. The tests were green because they tested a copy of the logic, not the call site.

A check that can’t fail isn’t a check. It’s a mutation away from a lie.

Mutation testing turned out to be the single best thing the verifier did. Not reading the diff, not running CI. Breaking the code on purpose and watching whether anything noticed.

Behaviour hides from screenshots

Screenshots caught real things. A spend badge squeezing the model name to “Claude S…”. Faded rows at 2.4:1 contrast. A button 21 px below the fold.

They never caught a lost click. Editing a session name and clicking another row saved the name and swallowed the click, because the blur re-rendered the list under the mouse. Pressing Enter right after typing in the command palette did nothing one time in three. Both looked perfect in every screenshot.

And the worst bugs only showed up with real programs. A CLI’s session file grew a 22 KB first line, past a hard-coded 16 KB read. Killing a child process left its own child holding the pipe open, so a “timeout” waited forever. One CLI reported its state only if the terminal answered a capability probe nobody documented — found by reading the binary, not the docs.

A unit test with a fake would have passed all three.

Screenshots prove layout. Only running the thing proves behaviour.

The check has an attack surface

Somewhere around PR thirty the verifier stopped being a QA step and started being a security review.

The desktop app showed git status for each session. git status is read-only, right. Except a repository can set core.fsmonitor to any command, and a .gitattributes filter can run another. Point the app at a cloned repo and it ran the repo’s code. The verifier proved it with a fixture that created a file called PWNED.

Clickable links accepted https://github.com@evil.com. A blocklist of executable extensions missed an executable with no extension, which macOS happily opens in Terminal.

None of the implementers thought of these. None of them were asked to. The PR said “show the git branch”, and it did.

The more you trust the loop, the more the check has to think like an attacker.

The author is a group of units

So far everything is about one PR. Here’s the part I haven’t solved.

When there’s one author, me, I don’t need to think about authors. When there are six agents and occasionally a human, the same question keeps coming back: how much should I trust this PR before I’ve checked it?

The honest answer is: it depends on who wrote it, and “who” should mean their record. An author is a group of PRs. Their credit should be a summary of how those PRs went.

CREDIT IN    merged, and still there 30 days later
CREDIT IN    the verifier tried to break it and couldn't
CREDIT OUT   reverted
CREDIT OUT   a follow-up fix to the same lines within a week
CREDIT OUT   each round of rework the verifier sent back
CREDIT OUT   a claim in the PR the evidence didn't support

Then credit buys one thing, and only one: how deep the check goes. High credit on a reversible, contained change gets the light gate. Low credit, or anything touching the irreversible lane, gets everything. It’s the blast-radius rule from On Harness with a second axis.

                    low blast radius       high blast radius
  high credit       targeted tests         full suite + review
  low credit        full suite + review    full suite + review + human

Credit never skips the gate on the irreversible lane. It just decides how expensive the reversible lanes are.

What open source already knows

None of this is new. Big projects have been managing too many PRs from too many people for twenty years.

And one number I keep coming back to. The .NET team ran a coding agent on dotnet/runtime for ten months. 878 PRs. 67.9% landed, against 87.1% for staff and 79.7% for the community. Revert rates were basically the same: 0.6% against 0.8%. But more than half of the agent’s merged PRs needed a human to push commits before they landed.

So the agent’s work, once merged, held up as well as anyone’s. The cost wasn’t in quality. It was in the rework nobody counts.

METR found the other half of it. About half of SWE-bench-passing patches wouldn’t be merged by the actual maintainers. Passing the tests and being mergeable are different checks.

A green suite is admission, not credit.

The inversion

Every system above was built for a world where PRs are scarce and reviewers are scarcer.

The ladders take months because trust takes months to observe. Ten PRs over a quarter is a lot of signal for a human. An agent can open ten PRs before lunch.

So the old instruments break in both directions. Time-based ladders are far too slow for agents. Volume-based ones are trivially farmed. Counting merged PRs means nothing when merging is cheap.

What’s left is the record per PR: how far a mutation got before a test noticed, how many rounds of rework, whether it was still there a month later. That’s what makes credit per PR, not per calendar.

And the reviewer is now the scarce resource on both sides. A verifier agent can run all day, but my attention can’t, and neither can the usage budget that runs out in the middle of a Tuesday.

But

I don’t know how to compute this yet. A few things I’m stuck on:

Credit for whom. If a PR goes bad, is it the agent’s, the model’s, the prompt’s, or mine for writing a vague issue? Most of the reworks traced back to the issue, not the code. My credit should probably drop with the PR’s.

Goodhart is right there. The moment “the verifier couldn’t break it” is worth credit, an implementer learns to write code the verifier can’t break, which is not the same as correct. The verifier needs its own record, against what actually got reverted later.

The noise floor is high. Some follow-up fixes are corrections and some are just the next feature. A revert can be the author’s fault or a product decision. Counting either without context punishes the wrong thing.

I made the mistakes too. I deleted a branch while its PR was still open, which closed it. I ran ten agents at once and burned a week of budget in a day. I reported a “pass” that was only an agent’s summary of someone else’s pass. If I were scored like the agents, my credit would have dropped.

So for now credit is a column in a spreadsheet, not a gate.

From here

Day zero needs a loop. Day thirty needs a ledger.

The PR is the unit. The author is a group of units. The check is the only thing that knows the difference.

Sources