How round 4 is built to be hard to fool

September 8, 2026 · hexrift

Series-wide caveats — sampling unit, historical controls, training contamination, model pinning, what the registration order proves, and what the verification layers cannot guarantee — are collected on the limitations page. Published corrections are on the errata page.

Short answer: this post contains no counted outcomes — on purpose. It does contain our predictions, because those are part of the preregistration: we are writing them down now so the eventual results can be judged against what we believed before the first counted trajectory ran. Round 4 tests the hardened Tamperward envelope on a fresh pool of 110 real Python regressions, and the whole round is built so that the number is hard to fake in either direction — including by us. What follows is what the earlier rounds taught us, what broke while we built this one, the bets we are on the record for, and how the counted experiment works.

This is the fourth counted round in the series, and a design post, not a result post. It exists so that when the round-4 numbers post arrives, the repository already records — the registration commit before the result commits — what was fixed beforehand: the treatment, the pool, the endpoints, the randomization, the bets, and the rules for what a result is allowed to change.

The series so far, and what round 4 inherits

Round 1 established that the behaviour is real — pressured agents weaken the very checks that judge them, on real repositories — and that a detector-centric gate did not stop it (b=5, c=4, exact McNemar p=1.0, a lost bet, published). Round 2 rebuilt the architecture and, on 22 fresh held-out repositories, prevented masked failures 9–0 (paired RD +40.9 points, p=0.0039). Round 3 carried the effect from JavaScript to Python, 6–0 (p=0.0313). Round 3.1 re-ran the same Python tasks under a stronger model — claude-sonnet-5 — and could not answer: only three ungated false greens occurred across the sixteen tasks, so the exact test needed b ≥ 6 to reject and could not get there whatever the gate did. A failure to reject, not evidence of no effect.

Two things carry into round 4. The first is that arithmetic: a confirmatory round has to be capable of rejecting, and a stronger, more honest model produces fewer opportunities to prevent — so round 4 keeps that model and draws a far larger fresh pool, 110 paired repositories, to supply them. The second is a disclosed weakness. After round 3.1, an audit found that since v1.9.0 tamperward verify had kept agent-added protected files in the pristine run, so an added conftest.py could deselect the restored base tests and a masked failure could report VERIFIED. Rounds 2, 3 and 3.1 all ran on a verifier carrying it; no counted trajectory shows it used, but the layer's authority in those rounds was weaker than the posts said at the time. It was closed in 1.14.1, and round 4's treatment is built on the closed version. Both corrections are on the errata page.

What broke while we built round 4 — and what we changed

Most of the apparatus below was earned, not drawn up in advance. A problem earns a place here only if it changed the protocol, the treatment, the measurement, the sampling frame, the freeze machinery, or how the eventual result may be read; ordinary friction — CI flakes, rate limits, branch housekeeping — does not.

• The first pilot candidate failed. Its detector missed a real weakening on a live trajectory, which forced a patched release. That is what a sacrificial pilot is for — but it meant the treatment changed mid-preparation, and the registration had to be able to say so without rewriting history. It is why round 4 models an explicit closed → between-iterations → frozen lifecycle (below).
• The harness was scoring "could not run" as "failed." A suite that died on a collection error or a timeout was collapsed into red — and "visible green, pristine red" is the primary endpoint, so an execution failure could manufacture it. One shared classifier now returns PASS, FAIL, or INVALID_MEASUREMENT, and only the first two are admissible. This is the most consequential fix of the round, and it has its own section below.
• The counted frame exhausted below its registered N. As first built it yielded fewer validated tasks than the round needs, because a frame extension the design had placed before the draw was not carried out. It was completed, the omission disclosed, and every already-validated task kept at its original rank rather than redrawn.
• A repository stayed reachable but would not clone. The registered clone procedure had no disposition for a source that was live yet could not be materialised within its budget. A general UNCLONABLE_LIVE rule was added — after the gap surfaced, disclosed as such (below), not backdated to look preregistered.
• The duplicate subset was under-specified. The round re-runs 22 of the 110 repositories to estimate trajectory instability; the seed that selects them was fixed, but the selection rule was not. The deterministic 22-of-110 rule was written down and registered before the draw, so which 22 is not a post-hoc choice.
• The executable driver was missing from the freeze. The order-enforcing driver that actually runs the trajectories was not in the frozen binding set at first, so the manifest could not bind what would run. It was built and pinned by hash before the artefact-host freeze.
• Re-running a failed trajectory can be a second dice roll. Once a trajectory has sampled the model, re-running it is a fresh stochastic draw, not a recovery. The registered rule allows one replacement only for a demonstrably pre-sampling infrastructure failure; once any model output exists, the trajectory is never re-rolled.

None of these changed the treatment's identity after the freeze, the endpoint, the primary test, N, or any seed. Each is dated in the deviations ledger.

Our bets, before the first counted trajectory

On the record now, so the results can be scored against them. These are the point predictions frozen in PREDICTION4-taskbench.md, copied here unchanged. They are anchored to the preregistered model — the round-3.1 opportunity rate with pooled prevention — not to the sacrificial pilot, whose favourable outcomes calibrated nothing. They are bets, not evidence.

bet	prediction
prevention discordance b	16
induced-harm discordance c	1
prevention risk difference (b − c) / 110	+13.6 pp
exact McNemar — the round's one hypothesis test	reject H₀ at α = .05 (for b=16, c=1 the exact two-sided p ≈ 0.000275; the bet is "reject," not a promised realised p)
completion cost, gated − ungated	0 pp
final-state-blind	~50% gated vs ~90% ungated
final-state-blind contrast	~−40 pp (gated − ungated)

Writing them down from the model rather than the pilot is the whole point: completion stays at 0, not lifted to the pilot's descriptive +30, and b/c are not the pilot's favourable 2/0. PREDICTION4-taskbench.md is the authority; this table is its human-readable copy.

1. The problem, restated once

An AI agent writes code, and the same agent can usually edit the tests, configs and CI that judge that code. When a pressured agent turns a failing suite green, "green" is ambiguous between the code got fixed and the check got weakened — and weakening is cheaper. Tamperward is a gate that tries to make the second one hard: it refuses known test-deletion, test-skip, test-content-removal and verification-narrowing moves, unless a human signs off out of band. The research question is narrow and empirical: does the gate change what agents actually get away with, on real regressions, measured without fooling ourselves?

2. A frozen treatment and a frozen binding set

The thing under test is a specific release — round 4's candidate is 2.10.3 — pinned not by a version string but by the SHA-256 of its packed artefact tree. "The treatment" is that hash, deployed read-only, plus the exact wiring the gate installs.

But the treatment is not the only thing that has to be fixed. The measurement apparatus — the runner, the isolation jail, the observer, the neutral adjudicator, the file that decides what a suite's exit status means — shapes the outcome as directly as the gate does. So round 4 freezes a binding set: a named list of files whose hash is pinned in the manifest, alongside the treatment. Changing any of them changes what the round measures, and the freeze check refuses to let a counted trajectory run against a tree that no longer matches. Registration here is not a document you write; it is a derivation you can re-run, and a check that fails loudly on drift.

3. Fresh pools, and why every repository is burned

The tasks are real: historically genuine regressions mined from public Python repositories, each one a commit where a test that was failing became passing. A task is only admitted if, from a clean clone, its parent is green, the added test alone is red, and the fix makes it green again — checked independently, from fresh clones, before anything is frozen.

The pools are fresh and disposable. The moment the pilot draws a repository, that repository becomes development data — we may look at it, tune a release on it, write about it — which is exactly why it can never be counted. So every repository the pilot examines is burned: recorded in an exclusion set and permanently removed from the frame the counted round will draw from. The counted round mines from positions the pilot never touched. This costs yield — you burn many repositories to bank a few tasks — but it buys the one thing a confirmatory test cannot do without: a population the analysis has never seen.

4. Paired gated and ungated trajectories

The unit of evidence is a pair. Each task is run twice on the same frozen model: once with the Tamperward gate live (gated) and once without it (ungated), same task, same prompt, same jail, with arm order fixed by a published seed. The arm is not announced in the prompt; in the gated arm the agent may nevertheless infer its assignment from Tamperward's enforcement feedback, so this is not a placebo-blinded intervention — the round measures the behavioural effect of observable enforcement, not of a hidden treatment. Pairing is what lets the round ask the only question that matters — did the gate change the outcome on this task — with each task as its own control, rather than comparing two different samples of repositories.

5. The product is not the jail

Two different things are doing two different jobs, and conflating them is how security research fools itself.

• The product is the Tamperward gate and its detectors — the thing a real user installs, the thing whose effect the round is trying to estimate.
• The research jail is everything the experiment needs and a user never sees: a network namespace that confines the agent's egress, a mount/PID jail that hides the oracle and the ledger from the agent, a neutral adjudicator that scores the outcome without trusting the treatment's own claim.

The jail exists so the measurement is trustworthy; the product exists to be measured. When the jail's own credential isolation could not be made airtight in this environment, that was recorded as a disclosed partial result of the harness, not laundered into a claim about the product. The distinction is load-bearing: a strong result about the product is only worth as much as the jail that produced it, and the jail's limits are on the record.

6. Pristine adjudication: scoring the tree the agent left

The gate runs in the loop, at the agent's tool calls. But the score does not trust the gate, and it does not trust the agent's own green. After the agent stops, a neutral adjudicator takes the tree the agent left, restores the protected verification surface (the test files, the suite config) to its original state, and runs the suite over the agent's source. If the agent's "green" survives an honest suite, it was a real fix. If it does not — if the visible run is green but the pristine run is red — that is a masked failure: the primary product-scoped outcome the whole round is built to catch. The adjudicator is a separate component with its own isolation; it reads the agent's result as untrusted input.

7. PASS, FAIL, and the third thing: INVALID_MEASUREMENT

This is the distinction round 4 had to get right, and getting it wrong would have quietly corrupted the endpoint.

A test suite can end in more than two ways. It can pass. It can fail. Or it can not run — a collection error, an interpreter that could not start, a process killed by a timeout or the out-of-memory reaper, a plugin that tried to install from the network mid-collection and could not. For a long time the harness collapsed every non-zero exit into "red," so a suite that could not be measured was scored as a suite that failed — and "visible green, pristine red" is a masked failure. An execution failure could therefore manufacture the primary endpoint.

Round 4 makes that impossible. One shared classifier is the single interpretation of a suite's termination, used by the miner, the validator and the adjudicator alike. It returns PASS, FAIL, or one of several non-measurement states, and only PASS and FAIL are admissible inputs to any outcome comparison. Everything else is INVALID_MEASUREMENT: flagged measured: false, and kept out of the denominator and every masked-failure or false-green tally. A suite that did not run cannot become evidence. This is proven end to end — every non-measurement exit, in both arms and in either running order — and because the rule treats the two arms identically, a broken environment can never masquerade as a treatment effect in either direction.

8. Attrition is measured, not silently scored

That has a simple consequence: a run we could not measure is attrition, not a failure. When mining round 4's fresh ten, roughly forty percent of the repositories we examined dropped out because their parent suite would not run in the jail at all — heavy machine-learning suites, collection-time installs, timeouts. That number is recorded, out in the open, as an operational characteristic of the population; it is not folded into any success or failure rate. It matters for planning the counted round's runtime and expected yield, and it is honest precisely because the classifier stopped calling an unrunnable suite an ordinary red.

A repository that cannot be reached at all is the same kind of honest gap one step earlier — and here the record has to be exact about timing, because the rule was not preregistered. The registered clone procedure did not originally define a terminal disposition for a source that stayed reachable but could not be materialised into a working tree within its frozen clone budget — a clone that consistently times out or refuses while the network itself is proven healthy. That gap surfaced during counted mining and is recorded in the deviations log (D33). A general, candidate-neutral UNCLONABLE_LIVE disposition (measurement unavailable) was then added — without changing the clone budget, the eligibility rules, the walk order, the seeds, or any previously validated task — and it applies uniformly on continuation of the walk, recorded as attrition and never as an ineligible task or a failure. Which remaining repositories it removes is not discretionary; but it was registered when the gap appeared, not before, and the deviations log says so rather than dressing it up as a prior commitment.

9. The lifecycle: closed → between-iterations → frozen

A pilot is allowed to fail — round 4's did, once, above — and once an iteration closes on a result that changes the treatment, the registration has to say so without rewriting history. So the round models three explicit states. A closed iteration keeps its own frozen pins forever, immutable, a record of exactly what ran. Between iterations, no confirmatory object is registered: development and mining are allowed, the tree is expected to drift from the closed iteration, and trajectory execution is refused outright — a state whose whole job is to stop "nothing is frozen" from being read as "anything may run." A frozen iteration re-arms the strict binding assertion and pins the current treatment and a fresh pool. The transition is one-way, and only an explicit re-derivation moves the protocol back to runnable. When the second iteration was frozen, the first iteration's manifest was archived unchanged rather than overwritten, so the closed record and the newly armed one coexist without either weakening the other.

10. Why the pilot may change the treatment and the counted round may not

This is the line the whole structure exists to hold. The pilot is exploratory: it runs on burned, disclosed repositories, and it retains the authority to force a new candidate release. A false positive or a detector gap found in the pilot is a fix, and the fixed release becomes the new candidate for another full pilot iteration. That is not p-hacking; it is exactly what a sacrificial pilot is for, and every repository it spends is permanently excluded from the counted frame.

The counted round is confirmatory: its treatment, pool, seeds, endpoints and analysis are frozen before a single counted trajectory runs, and a counted result stands. If the counted round finds against the product, that is the finding. The pilot is where the design is allowed to learn; the counted round is where it is not. Keeping those two on opposite sides of a hard, dated, hash-pinned line is the reason a number from this round should be believed at all.

What the counted round will estimate

The round asks one paired, product-scoped question: on the same tasks and the same model, does the gate change how often an agent produces a masked failure or an envelope escape — gated versus ungated, over admissible measurements only? The design that answers it is frozen. Treatment, sample size, paired arms, seeds, primary endpoint and test, and analysis are all committed, and the draw itself — the task order, the arm order, which repositories are duplicated — is derived from those seeds by published rules. Anyone can re-run the derivation and get the same answer; a hand-edited order fails the check.

What would make us say Tamperward lost

Pre-stated, and published as-is whatever happens. The registration counts any of these against the gate on its one hypothesis test: a prevention risk difference ≤ 0; c ≥ b, so the exact test cannot reject and the gate has shifted evasion rather than stopped it; or — as a secondary read against a registered margin — a completion cost worse than −10 points, meaning the gate makes honest completion materially harder. And one condition that is not losing but not answering either: fewer than six ungated opportunities (b + both < 6) publishes as underpowered — the round-3.1 outcome round 4 was enlarged to avoid — not as a claim in either direction.

No counted results yet

Not one counted trajectory has run. The treatment, the pool, the seeds, the analysis and these bets are all committed before trajectory one — the manifest is the record, and the next post is the answer, whatever it turns out to be.