The paraphrase
The rule was simple. Some actions an agent must never take on its own: merge now and verify later, delete anything under ~, push to someone else’s repo. Anything that looks like one of those goes to a human.
So the gate scanned the agent’s note for a list of risky phrases. It worked for a day.
Then an agent wrote “merge now, verify later” as a note whose first character was a non-breaking space. The parser looked for decided: at the start of a line. It found a space-shaped character and moved on. The verifier failed the PR. I fixed it. The next PR failed on a different paraphrase.
Third time, I stopped patching the list.
The slide
In On Harness I said a loop needs a check that can fail. In On Units I said the PR is the thing that gets checked, and the verifier should try to break it.
Both are true. They’re also the easy part.
The part I skipped: the gate is software too. It has inputs, it has parsers, and the thing it’s guarding is very good at producing inputs the parser didn’t expect. An agent isn’t adversarial. It just writes in a different phrasing every time, which is the same thing from the parser’s side.
A sentence is not a fact
Every leaky gate this week had the same shape. It read prose and decided what the prose meant.
LEAKS regex over the agent's note for risky words
LEAKS a deny-list of phrases that mean "ask first"
LEAKS "claims verified" extracted from a PR body
LEAKS an allow-list matched by word ("clean ~ and /" passed "clean")
HOLDS a typed action the CLI knows: merge, delete, push
HOLDS preconditions the CLI checks from the world, not the note
HOLDS a structured declaration with fields, not a paragraph
The fix wasn’t a better regex. It was changing what counts as input.
An action the gate can decide on its own is now a type. merge carries its preconditions: a verifier pass recorded at this exact commit, the repo is mine, the PR touches only what the task owned. The CLI reads those from state. If one is missing the answer is no, and nothing about the agent’s wording changes that.
Prose still exists. It’s advisory, read by the LLM verifier, who is free to be unimpressed.
Fail closed on facts. Let prose be a hint.
Break the gate
The verifier I trust most does one thing the others don’t. It mutates the code and watches whether the tests notice. I wrote about that in On Units. This week it became a tool, orbit mutate --diff: one mutant per changed statement, in a throwaway worktree.
The guard scripts were the worst offenders. One bash gate got a new surviving mutant on every verify round. Four rounds. Each time the tests were green and the guard was wrong in a way nobody had written a test for.
Surviving mutants now become TEST-GAP follow-ups instead of arguments. If a mutant lives, either the code is dead or the test is missing. There’s no third option, and “equivalent” needs a probe, not a shrug.
The gate got honest
Here’s the number that surprised me. Before the verifier had a fixed rubric, 82% of PRs passed on the first try. After, 19%.
tasks rework first pass
before 0.13.0 304 12% 82%
0.15.1 38 77% 19%
The agents didn’t get worse. The gate stopped being polite. A fixed /10 rubric with hard fails for a regression, a surviving mutant, or a claim the evidence doesn’t back turned “looks fine” into “show me.”
Small sample on the second row, and the tasks aren’t identical. I’d still rather read 19% than 82%. The 82% was a gate agreeing with itself.
An adversary from another family
The verifier and the implementer were the same model family. That’s two optimists agreeing, with extra steps.
So I added an adversary before the verifier, and for medium and hard tasks it runs on a different family. Across the logged runs it filed 43 findings on the hard tasks and 37 held up when I replayed them. That’s the cheapest gate I have per confirmed bug.
It also rewrote my credentials.
The first real run went through a throwaway HOME that I’d symlinked back to the real config, so the tool could log in. The tool refreshed its token and wrote through the link. One provider’s login gone, another half-gone. The tests had passed, because the tests used a fake binary, and a fake binary can’t overwrite anything real.
Lesson I should have known: a test with a fake third-party tool can’t prove credential safety. Run the real binary on a copy. Never link the owner’s secrets into a sandbox.
The cross-family runner now goes through a separate terminal host and is off by default.
But
The yield table has a column called dismissed. It’s 0 in every row.
That’s not a good sign. It means the log records findings that got fixed, and has nowhere to put the ones that were wrong. I can say 37 of 43 held up because I counted them by hand. The tool can’t say it yet. A gate with no false-positive column is a gate nobody is checking.
Also: I’m not sure it’s cheaper. The hard tasks cost roughly 470 agent-minutes per PR. Most of that is the gates arguing with the implementer. They earn it on a security fix. On a one-line copy change they’d be absurd, so the routing between “light” and “everything” is doing a lot of quiet work, and I don’t have a metric for it yet.
And a fair chunk of this week was one more family of agents checking the last family’s work. At some point somebody has to be the base case.
For now it’s a short list of facts the CLI can read, and a person at the end.
The gate is code. Check it like code.